vLLM Explained

What you need from DeepInfer Explained

BeginnerLearn the basicsabout 8 min read + 1 min deep dives

Code version: vLLM @ 3e44c0d · 2026-10-22

In one sentence

This site builds on the 12 basics pages of DeepInfer Explained; this page recaps each one in one line and says what vLLM does differently.

The idea#

DeepInfer Explained is a sister site. Its part "Learn the basics" teaches the general ideas of LLM inference: tokens, attention, the KV cache, batching and more. These ideas are the same in every engine. This site does not explain them again.

Use this page in two ways:

  • As a check. Read the recap lines. If a line is not clear to you, read the DeepInfer page first. Each link opens the DeepInfer site in a new tab.
  • As a map. The last column says where vLLM is different from DeepInfer. Each line cites the vLLM code and names the page of this site that explains it.

An everyday comparison. Think of a second car from a new maker. You already know how to drive. You only need to learn where this car puts the controls, and what it does in a different way. The comparison stops here: vLLM also adds parts that DeepInfer does not have at all, for example prefix caching and an OpenAI-compatible server.

Two facts touch every row of the table. First, vllm serve runs an API server process and a separate engine core process (vllm/v1/engine/async_llm.py:188-193). With one GPU, the worker runs inside the engine core process (vllm/config/parallel.py:1099-1100). Second, Model Runner V2 runs the model by default (vllm/config/vllm.py:730-748).

The 12 basics, and what vLLM changes#

#DeepInfer pageRecap (one line)What is different in vLLM
1What an LLM does An LLM gives a probability to every possible next token; an engine runs it in a loop: pick a token, add it, run again. One EngineCore.step() schedules, runs the model and updates all requests of the batch (vllm/v1/engine/core.py:657-686). Async scheduling is on by default when the configuration allows it (vllm/config/vllm.py:1664-1698). Then the engine core uses step_with_batch_queue(), which schedules the next batch before the current batch ends (vllm/v1/engine/core.py:227-229, vllm/config/vllm.py:591-593). See The engine core loop: threads, queues and the busy loop.
2Tokens and tokenizers A tokenizer cuts text into tokens from a fixed vocabulary and gives each token an integer id. The API server applies the chat template and tokenizes before the engine core gets the request (vllm/renderers/base.py:1385-1396). Detokenization is incremental, in the output processor of the front end (vllm/v1/engine/output_processor.py:307-310, vllm/v1/engine/async_llm.py:180). See The OpenAI-compatible API.
3Inside a transformer Embedding, then a stack of layers (attention + MLP, with norms and residuals), then one score per vocabulary token. Most model families have one Python file in vllm/model_executor/models/. A registry maps the architectures name of config.json to a file and a class. Several names can share one class: InternLM3ForCausalLM also uses llama.py (vllm/model_executor/models/registry.py:140, vllm/model_executor/models/registry.py:152). The layer stack is in classes such as LlamaDecoderLayer (vllm/model_executor/models/llama.py:251). See How a vLLM model file is written (llama.py).
4Attention, step by step Each token scores its query against earlier keys, applies softmax, and mixes the values; GQA and MLA make the KV cache smaller. Most models use the Attention layer class (vllm/model_executor/layers/attention/attention.py:229); MLA models use MLAAttention (vllm/model_executor/layers/attention/mla_attention.py:426). vLLM selects an attention backend (FlashAttention, FlashInfer, ...) from the head size, the dtype, the KV cache dtype, MLA and other layer properties (vllm/v1/attention/selector.py:109-122). See Attention backends: backend, metadata builder and implementation.
5Prefill and decode Prefill reads the whole prompt in one pass (compute-bound); decode adds one token per step (memory-bound). The scheduler has no prefill phase and no decode phase. One step can mix prompt chunks and decode tokens (vllm/v1/core/sched/scheduler.py:624-633). See The V1 scheduler: one token budget, no phases.
6The KV cache The engine keeps the keys and values of earlier tokens in fixed-size blocks, with a block table per request. Blocks have 16 tokens by default; the attention backend can select another size (vllm/config/cache.py:182, vllm/platforms/interface.py:771-781). Prefix caching is on by default for most generative models (vllm/config/cache.py:254, vllm/engine/arg_utils.py:3062-3063, vllm/config/model.py:2275-2291). See Prefix caching: the idea.
7Sampling: temperature, top-k, top-p Sampling turns logits into probabilities, removes unlikely tokens (top-k, top-p) and draws one token with a seeded random number. Model Runner V2 samples on the GPU. In the general path, it applies top-k/top-p, then draws with a Gumbel sample from the request seed and the token position (vllm/v1/worker/gpu/sample/sampler.py:347-358). A request without a seed gets a random seed (vllm/v1/worker/gpu/sample/states.py:54-58). On supported hardware, a fused FlashInfer (or XPU) sampler can replace this path (vllm/v1/worker/gpu/sample/sampler.py:339-345). It does so only when requests use top-k or top-p, no request is greedy or has a seed, and no request needs processed logprobs (vllm/v1/worker/gpu/sample/sampler.py:304-315). See Sampling and logits processors in Model Runner V2.
8Batching Continuous batching makes a new packed batch in each step; chunked prefill splits a long prompt over steps. The scheduler serves running requests first, then waiting ones, in one token budget (vllm/v1/core/sched/scheduler.py:713-715, vllm/v1/core/sched/scheduler.py:956). Chunked prefill is on by default for generative models that are not encoder-decoder models (vllm/engine/arg_utils.py:3031-3035, vllm/config/model.py:2229-2236). See Serving metrics: TTFT, TPOT, ITL, throughput, goodput.
9GPUs, kernels, and speed A GPU runs kernels on many cores; speed depends on memory traffic, math and launch cost; CUDA graphs remove launch cost. vLLM compiles the model with torch.compile at every optimization level above O0 (vllm/config/vllm.py:1806-1810). The default level O2 sets cudagraph_mode to FULL_AND_PIECEWISE (vllm/config/vllm.py:434, vllm/config/vllm.py:310). See torch.compile and CUDA graphs in vLLM.
10Parallelism basics Work splits by requests (DP), matrices (TP), layers (PP), tokens (CP) or experts (EP); collectives move the data. vLLM has TP and PP groups; DeepInfer has neither. The rank grid is (-1, DP, PP, PCP, TP) (vllm/distributed/parallel_state.py:2157-2163). See Tensor and pipeline parallelism in practice.
11Mixture of Experts (MoE) A router sends each token to a few of many expert MLPs, so each token uses only a small part of the weights. MoE layers come from the function FusedMoEFactory (vllm/model_executor/layers/fused_moe/layer.py:92). Expert parallelism is off by default; then MoE layers use tensor parallelism (vllm/config/parallel.py:190-191). See Expert parallelism and MoE serving.
12Number formats: BF16, FP8, FP4 A format sets the bits for sign, exponent and mantissa; FP8 and FP4 use block scales to keep accuracy. vLLM loads many quantized checkpoint formats: FP8, AWQ, GPTQ, compressed-tensors, MXFP4 and more (vllm/model_executor/layers/quantization/__init__.py:15-49). The KV cache dtype auto uses the model dtype for most models (vllm/config/cache.py:232-240). See Quantized checkpoints.

Deep dive#

Deep dive

Defaults that make vLLM numbers differ from DeepInfer numbers

The DeepInfer pages use DeepInfer's settings in their examples. These vLLM defaults change the numbers. Each row cites the vLLM line.

SettingvLLM defaultSource
KV block size16 tokens (DeepInfer: 128). A backend can select another size when you do not set --block-size.vllm/config/cache.py:459-460, vllm/platforms/interface.py:771-781
Prefix cachingon for most generative models (not encoder-decoder or attention-free models); block hash sha256vllm/config/cache.py:254-256, vllm/engine/arg_utils.py:3062-3063, vllm/config/model.py:2275-2291
Chunked prefillon for generative models (not encoder-decoder)vllm/engine/arg_utils.py:3031-3035
Step kindsmixed: prompt chunks and decode tokens in one stepvllm/v1/core/sched/scheduler.py:624-633
Token budget for vllm serve on a GPU2,048, 8,192 or 16,384 tokens, by GPU memory. --performance-mode throughput doubles the value.vllm/engine/arg_utils.py:2960-2993, vllm/engine/arg_utils.py:3171-3175
max_num_seqs for vllm serve on a GPU256 or 1,024, by GPU memoryvllm/engine/arg_utils.py:2960-2993
Preemptionfree all blocks of the request, set its computed tokens to 0, put it at the front of the waiting queuevllm/v1/core/sched/scheduler.py:1676-1700
CUDA graphsFULL_AND_PIECEWISE at the default optimization level O2vllm/config/vllm.py:310, vllm/config/vllm.py:434
Expert parallelismoffvllm/config/parallel.py:190-191

Worked example. A request with a prompt of 1,000 tokens needs ⌈1,000 / 16⌉ = 63 blocks in vLLM with the default size. With DeepInfer's 128-token blocks, it needs ⌈1,000 / 128⌉ = 8 blocks. The memory is almost the same; only the number and size of the blocks change.

What can go wrong#

  • The DeepInfer code is not the vLLM code. The "In DeepInfer" parts of the DeepInfer pages cite DeepInfer files. Use them only for the idea. For vLLM behaviour, use the citations on this site.
  • DeepInfer's step kinds do not apply. A DeepInfer step is a prefill step or a decode step. A vLLM step can be both (vllm/v1/core/sched/scheduler.py:624-633). Do not expect separate prefill and decode step times in vLLM metrics.

Why it matters. The rest of this site uses the basics without a new explanation. A gap in the basics makes the concept pages hard to follow.

Check yourself#

  1. 1
    Which DeepInfer page explains block tables and preemption, and what is the default block size in vLLM?
    easy
    Show answer

    The KV cache. vLLM uses 16 tokens per block by default (vllm/config/cache.py:182). The attention backend can select another size when you do not set one (vllm/platforms/interface.py:771-781).

  2. 2
    A DeepInfer step is a prefill step or a decode step. Can one vLLM step hold decode tokens of 50 requests and a chunk of a new prompt?
    medium
    Show answer

    Yes. The vLLM scheduler has no prefill or decode phase. It gives tokens to each request until its computed tokens catch up (vllm/v1/core/sched/scheduler.py:624-633). It serves the running requests first, then the waiting ones with the rest of the token budget (vllm/v1/core/sched/scheduler.py:713-715, vllm/v1/core/sched/scheduler.py:956).

  3. 3
    You start vllm serve on one H100 with 80 GB and set no scheduler flags. What are the token budget and max_num_seqs?
    hard
    Show answer

    8,192 tokens and 1,024 requests. vllm serve uses the usage context OPENAI_API_SERVER (vllm/entrypoints/launchers/api_server/entry.py:72, vllm/entrypoints/launchers/api_server/entry.py:82). An H100 with 80 GB reports about 79 GiB (inferred, not in the repo). That is 70 GiB or more and less than 160 GiB, and the GPU is not an A100 or a GB10. So the "H100 and H200" defaults apply (vllm/engine/arg_utils.py:2970-2983). This is true with the default --performance-mode; the mode throughput doubles both values (vllm/engine/arg_utils.py:3171-3175).

  • What vLLM is todayvLLM is an inference and serving engine for LLMs. At this commit it has one engine, V1. You use it through vllm serve (an OpenAI-compatible HTTP server, the main path of this site) or through the offline LLM class. Both reach the same engine core.
  • Serving metrics: TTFT, TPOT, ITL, throughput, goodputServing metrics tell how fast a server answers one user (TTFT, TPOT, ITL, E2E latency) and how much work it does for all users (throughput, goodput). vLLM measures them in the API server and the engine core, and vllm bench serve measures them at the client.
  • Prefix caching: the ideaChat requests often start with the same tokens. Prefix caching keeps the KV cache blocks of earlier requests under a chain of hashes, so a new request with the same prefix skips the prefill of those blocks and gets a shorter TTFT.
  • Tensor and pipeline parallelism in practiceTensor parallelism splits every weight matrix across GPUs and adds the parts with an all-reduce. Pipeline parallelism gives each GPU a range of layers. This page shows the split of one MLP with real shapes, the head limits of attention, and how to choose TP, PP, DP or EP for a deployment.
  • The V1 scheduler: one token budget, no phasesIn each engine step, Scheduler.schedule() shares one token budget: first among the running requests, then among the waiting ones. It has no prefill or decode phase, it preempts the last running request when KV blocks run out, and update_from_output() appends the sampled tokens.
  • The KV cache manager: blocks, the block pool and memory sizingIn the engine core, the KVCacheManager hands out KV blocks to the scheduler through a coordinator, one single-type manager per KV cache group and one BlockPool. The workers only receive block ids. The number of blocks comes from a memory profile at start-up.
  • The model runner (Model Runner V2)Model Runner V2 keeps every running request in a fixed row of GPU tensors, gathers each step's inputs from those rows with GPU kernels, runs the model (CUDA graph or eager), and samples in a second call that returns an AsyncOutput without a GPU wait.