vLLM Explained

The model runner (Model Runner V2)

AdvancedvLLM conceptsabout 14 min read + 6 min deep dives

Code version: vLLM @ 3e44c0d · 2026-10-22

In one sentence

Model Runner V2 keeps each request in a fixed GPU row, builds every step's inputs on the GPU, runs the model, and samples without waiting.

The idea#

The worker owns one GPU, and the model runner does the GPU work of each engine step. It gets a SchedulerOutput that says which requests run and how many tokens each one adds. It must turn this plan into tensors, run the model, and sample the next tokens. The default runner is Model Runner V2 (MRV2) in vllm/v1/worker/gpu/.

MRV2 is built on three ideas:

  • A persistent batch. Each running request gets a fixed row, its request slot, in a set of tensors with max_num_seqs rows. The rows hold the token ids, the lengths and the last sampled token. They stay on the GPU between steps.
  • Inputs built on the GPU. Each step sorts the requests into a batch order. A small tensor, idx_mapping, maps each batch row to its slot. GPU kernels use it to gather the input ids, positions and lengths from the rows.
  • No wait for the GPU. execute_model only queues the forward pass. sample_tokens queues the sampler and a copy to the CPU, and returns an AsyncOutput. Somebody waits for the copy later, in get_output(). This is what makes async scheduling work (Async scheduling: overlap the CPU and the GPU).

An everyday comparison. Think of a hotel with numbered rooms. A guest keeps one room for the whole stay, and the luggage stays in that room. Each morning the front desk writes a short list: "visit rooms 1023, 1022 and 1021, in this order". The staff use the list to find the luggage; nobody carries it to the desk. The comparison stops here. In MRV2, a guest who is moved out (preempted) gives the room back and starts again as a new guest later.

The picture#

Figure: what MRV2 keeps on the GPU between steps (panel 1), how one step turns the scheduler plan into input tensors (panel 2), and what goes to the attention layers (panel 3). Values are a worked example with block size 16.
Open full size ↗

Read panel 1 as the long-lived memory, and panels 2 and 3 as the work of one call of execute_model. The amber arrow is the only input copy from the CPU in each step: idx_mapping and query_start_loc. Speculative decoding adds a few more small arrays. The staged writes of new requests and new blocks (Step 3) also go to the GPU. Everything else is read on the GPU, through the slot numbers.

In the code#

Step 1: why V2, and when V1 runs#

The design document lists the problems of the first runner. V1 used its persistent tensors directly as model inputs, so a finished request forced a reorder of whole tensors. It also kept a second copy of every request state (docs/design/model_runner_v2.md:21-27). MRV2 keeps the state apart from the per-step inputs and gathers the inputs on the GPU (docs/design/model_runner_v2.md:31-39). It assumes that the execution loop is "a CUDA stream with no CPU synchronization points" (docs/design/model_runner_v2.md:47).

The worker builds MRV2 when use_v2_model_runner is true, and Model Runner V1 (vllm/v1/worker/gpu_model_runner.py) only as the fallback (vllm/v1/worker/gpu_worker.py:567-583). The fallback runs without Triton or with an unsupported feature: PP with external_launcher, suffix or custom_class drafts, parallel EAGLE drafts, mixed-vocabulary drafts, some dual-batch-overlap cases (vllm/config/vllm.py:734-746, vllm/config/vllm.py:2999-3033). The rest of this site explains MRV2 only.

Step 2: the persistent state and the request slots#

The runner makes one RequestState with max_num_reqs = max_num_seqs rows (vllm/v1/worker/gpu/model_runner.py:286-295). A new request takes a free row:

vllm/v1/worker/gpu/states.py:91-118GitHub ↗
    def add_request(
        self,
        req_id: str,
        prompt_len: int,
        all_token_ids: list[int],
        num_computed_tokens: int,
        max_tokens: int,
    ) -> None:
        assert len(self.free_indices) > 0, "No free indices"
        req_idx = self.free_indices.pop()
        self.req_id_to_index[req_id] = req_idx
        self.index_to_req_id[req_idx] = req_id

        self.max_seq_len[req_idx] = prompt_len + max_tokens
        self.prompt_len.np[req_idx] = prompt_len
        prefill_len = len(all_token_ids)
        assert prefill_len >= prompt_len, (
            f"prefill_len {prefill_len} < prompt_len {prompt_len}"
        )
        self.prefill_len.np[req_idx] = prefill_len
        self.total_len.stage_write_elem(req_idx, prefill_len)
        self.all_token_ids.stage_write(req_idx, 0, all_token_ids)
        self.num_computed_prefill_tokens[req_idx] = num_computed_tokens
        self.num_computed_tokens_np[req_idx] = num_computed_tokens
        self.num_computed_tokens.stage_write_elem(req_idx, num_computed_tokens)

        if self.num_speculative_steps > 0:
            self.draft_tokens[req_idx].zero_()
  1. Lines 99-102. free_indices starts as [0, 1, ..., max_num_reqs - 1] and pop() takes the last one (vllm/v1/worker/gpu/states.py:30). So the first request gets slot 1023 when max_num_seqs = 1024.
  2. Lines 110-115. The writes are staged on the CPU. apply_staged_writes() later copies them to GPU-visible memory: a UVA copy for the small arrays, one kernel for each staged tensor (vllm/v1/worker/gpu/states.py:120-125).

remove_request appends the slot to free_indices again, so the slot freed last is used next (vllm/v1/worker/gpu/states.py:127-134). The runner treats a preempted request like a finished one: finish_requests removes both kinds in sorted order, so every TP rank frees the same slots (vllm/v1/worker/gpu/model_runner.py:1173-1185).

Step 3: apply the plan to the state#

Each execute_model call starts with the state changes of the plan:

vllm/v1/worker/gpu/model_runner.py:1753-1771GitHub ↗
        if not dummy_run:
            # Update the request states.
            self.update_pp_decode_requests()
            self.finish_requests(scheduler_output)
            self.free_states(scheduler_output)
            self.add_requests(scheduler_output)
            self.update_requests(scheduler_output)
            self.block_tables.apply_staged_writes()
            if self.aux_output_connector is not None:
                # Register this step before the GPU forward.
                self.aux_output_connector.begin_step(
                    scheduler_output.aux_output_connector_metadata
                )
            if scheduler_output.total_num_scheduled_tokens == 0:
                # No need to run the model.
                empty_output = self.kv_connector.no_forward(scheduler_output)
                return self._merge_ec_connector_no_forward(
                    scheduler_output, empty_output
                )

The block ids come from the scheduler's block pool; The KV cache manager: blocks, the block pool and memory sizing explains how they are chosen. The runner only mirrors them on the GPU.

Step 4: the batch order#

gather_batch_req_state sorts the scheduled request ids:

vllm/v1/worker/gpu/model_runner.py:2471-2483GitHub ↗
def sort_batch_req_ids(
    num_tokens_per_req: dict[str, int],
    draft_tokens: dict[str, list[int]],
    decode_query_len: int,
) -> list[str]:
    # Order verification/decode -> short_extend -> prefill;
    # split_decodes_and_prefills relies on decode-like requests leading.
    key = lambda r: (
        not draft_tokens.get(r),
        (num := num_tokens_per_req[r]) != decode_query_len,
        num,
    )
    return sorted(num_tokens_per_req, key=key)

The key puts requests with draft tokens first (speculative verification). Then come the requests whose token count equals decode_query_len, the decodes. All others follow, sorted by token count, so short extends come before long prefill chunks. The comment says why: split_decodes_and_prefills in the attention backends expects the decode-like rows at the front. For a request without speculative decoding, decode_query_len is 1 (vllm/v1/worker/gpu/model_runner.py:442-445, vllm/v1/worker/gpu/model_states/interface.py:254-256).

The same function computes idx_mapping_np, the slot of each sorted request, and decides whether the batch has a prefill (vllm/v1/worker/gpu/model_runner.py:1310-1326). It returns a uniform decode token count only for a batch that may use a decode CUDA graph (vllm/v1/worker/gpu/model_runner.py:1370-1372).

Step 5: prepare_inputs builds the step tensors on the GPU#

Only small arrays go from the CPU to the GPU: idx_mapping and query_start_loc. The rest is computed on the GPU:

vllm/v1/worker/gpu/model_runner.py:1446-1504GitHub ↗
        # Get query_start_loc.
        # num_reqs_padded is None for PIECEWISE graphs (no request padding needed)
        num_reqs_padded = batch_desc.num_reqs or num_reqs
        query_start_loc_np = np.empty(self.max_num_reqs + 1, dtype=np.int32)
        query_start_loc_np[0] = 0
        np.cumsum(num_scheduled_tokens_np, out=query_start_loc_np[1 : num_reqs + 1])
        # Pad for full CUDA graph mode.
        # Some attention backends like FA3 require query_start_loc to be non-decreasing.
        query_start_loc_np[num_reqs + 1 :] = num_tokens
        query_start_loc = self.input_buffers.query_start_loc
        async_tensor_h2d(query_start_loc_np, out=query_start_loc)
        if adaptive_verification is not None:
            cu_num_logits, query_start_loc, total_num_draft_tokens = (
                adaptive_verification.reallocate_drafts(req_ids, idx_mapping)
            )
            total_num_logits = num_reqs * num_bonus_tokens + total_num_draft_tokens
        if num_draft_tokens_per_req is not None:
            expanded_idx_mapping, expanded_local_pos = expand_idx_mapping(
                idx_mapping, total_num_logits, cu_num_logits, self.decode_query_len
            )
        query_start_loc_np = query_start_loc_np[: num_reqs_padded + 1]
        query_start_loc = query_start_loc[: num_reqs_padded + 1]

        # Get prefill tokens if any.
        if batch_req_state.has_prefill:
            prepare_prefill_inputs(
                self.input_buffers.input_ids,
                self.req_states.next_prefill_tokens,
                idx_mapping,
                query_start_loc,
                self.req_states.all_token_ids.gpu,
                self.req_states.prefill_len.gpu,
                self.req_states.num_computed_tokens.gpu,
            )

        # Prepare positions and seq_lens.
        prepare_pos_seq_lens(
            idx_mapping,
            query_start_loc,
            self.req_states.num_computed_tokens.gpu,
            self.input_buffers.positions,
            self.input_buffers.seq_lens,
        )
        seq_lens = self.input_buffers.seq_lens[:num_reqs_padded]

        # Some input token ids are directly read from the last sampled tokens
        # and draft tokens. Also, get the logits indices to sample tokens from.
        logits_indices = combine_sampled_and_draft_tokens(
            self.input_buffers.input_ids,
            idx_mapping,
            self.req_states.last_sampled_tokens,
            query_start_loc,
            seq_lens,
            self.req_states.prefill_len.gpu,
            self.req_states.draft_tokens,
            cu_num_logits,
            total_num_logits,
            self.model_state.num_new_sampled_tokens_per_step,
        )
  1. Lines 1449-1456. query_start_loc is the cumulative sum of the token counts, in a persistent buffer of max_num_reqs + 1 entries. Unused entries repeat the total, because "some attention backends like FA3 require query_start_loc to be non-decreasing".
  2. Lines 1470-1479. Only when the batch has a prefill: a kernel copies the prompt tokens from all_token_ids of each slot into input_ids.
  3. Lines 1482-1488. A kernel writes positions and seq_lens. For each request, seq_len = num_computed_tokens + query_len (vllm/v1/worker/gpu/input_batch.py:375-376).
  4. Lines 1493-1504. A kernel writes the last sampled token (and the drafts) of each decode into input_ids, and returns logits_indices: the rows that get logits. Without drafts, this is the last token of each request.

The varlen layout itself is the one of Input metadata. The result is an InputBatch dataclass with these tensors and their CPU twins (vllm/v1/worker/gpu/input_batch.py:38-125).

Step 6: block tables, slot mappings and attention metadata#

vllm/v1/worker/gpu/model_runner.py:1577-1597GitHub ↗
    def prepare_attn(
        self, input_batch: InputBatch
    ) -> tuple[tuple[torch.Tensor, ...], torch.Tensor]:
        if self.pcp_manager is not None:
            return self.pcp_manager.prepare_attn(input_batch)

        # Block tables: num_kv_cache_groups x [num_reqs_padded, max_num_blocks].
        block_tables = self.block_tables.gather_block_tables(
            input_batch.idx_mapping,
            num_reqs_padded=input_batch.num_reqs_after_padding,
            num_computed_tokens=self.req_states.num_computed_tokens.gpu,
        )
        # Slot mappings: [num_kv_cache_groups, num_tokens_padded].
        # Kernel pads beyond num_tokens with PAD_SLOT_ID.
        slot_mappings = self.block_tables.compute_slot_mappings(
            input_batch.idx_mapping,
            input_batch.query_start_loc,
            input_batch.positions,
            num_tokens_padded=input_batch.num_tokens_after_padding,
        )
        return block_tables, slot_mappings

gather_block_tables copies the rows of the batch's slots into a second set of tensors in batch order. compute_slot_mappings computes the slot mapping: one row per KV cache group, one entry per token. The entry is block_number × block_size + offset (vllm/v1/worker/gpu/block_table.py:409), and padding tokens get PAD_SLOT_ID = -1 (vllm/v1/attention/backends/utils.py:53).

Then the runner builds a dict from layer name to slot mapping (vllm/v1/worker/gpu/attn_utils.py:447-455), and the model state builds the attention metadata per layer (vllm/v1/worker/gpu/model_runner.py:1940-1951). Attention backends: backend, metadata builder and implementation explains the builders.

Step 7: pick a CUDA graph mode and run the model#

Before prepare_inputs, the runner asks the CUDA graph manager for a descriptor of this batch shape (vllm/v1/worker/gpu/model_runner.py:1806-1819). The answer is FULL, PIECEWISE or NONE, plus the padded token count. The rule is on torch.compile and CUDA graphs in vLLM. Then the forward pass runs in one of three ways:

vllm/v1/worker/gpu/model_runner.py:2025-2070GitHub ↗
            # Use explicit cudagraph replay for FULL mode.
            # NOTE(woosuk): Here, we don't need to pass the input tensors,
            # because they are already copied to the CUDA graph input buffers.
            assert self.cudagraph_manager is not None
            self.kv_connector.pre_forward(
                scheduler_output, input_batch, attn_metadata=attn_metadata
            )
            model_output = self.cudagraph_manager.run_fullgraph(batch_desc)
        else:
            # For piecewise and eager mode, just call model().
            batch_descriptor = BatchDescriptor(
                num_tokens=input_batch.num_tokens_after_padding,
                has_lora=self.lora_config is not None,
                num_active_loras=batch_desc.num_active_loras,
            )

            with set_forward_context(
                attn_metadata,
                self.vllm_config,
                num_tokens=input_batch.num_tokens_after_padding,
                cudagraph_runtime_mode=batch_desc.cg_mode,
                num_tokens_across_dp=dp_sync_state
                and dp_sync_state.num_tokens_across_dp,
                batch_descriptor=batch_descriptor,
                ubatch_slices=ubatch_slices,
                slot_mapping=slot_mappings_by_layer,
                skip_compiled=skip_compiled,
                is_padding=input_batch.is_padding,
            ):
                self.kv_connector.pre_forward(scheduler_output, input_batch)
                if ubatch_state is not None:
                    assert self.ubatch_runner is not None
                    model_output = self.ubatch_runner.run(
                        self.model, model_inputs, ubatch_state
                    )
                elif batch_desc.cg_mode == CUDAGraphMode.PIECEWISE:
                    # Run the PIECEWISE graph (compiled PW cudagraph or breakable
                    # cudagraph, chosen inside run_pw_graph). cg_mode is only
                    # PIECEWISE after the cudagraph manager exists.
                    assert self.cudagraph_manager is not None
                    model_output = self.cudagraph_manager.run_pw_graph(
                        self.model, model_inputs
                    )
                else:
                    # Eager (NONE): call the raw model directly.
                    model_output = self.model(**model_inputs)

FULL mode replays the whole captured graph; the inputs are already in its buffers. The other modes run inside set_forward_context, which gives the attention layers their metadata (The forward context: how layers find per-step data). The output is not returned. The runner stores it in execute_model_state and returns None (vllm/v1/worker/gpu/model_runner.py:2090-2110).

Step 8: sample_tokens#

Figure: one step of MRV2 over time. The CPU only queues work in execute_model and sample_tokens; the only wait is copy_event.synchronize() in get_output().
Open full size ↗

The engine core calls sample_tokens after execute_model, with the grammar bitmask or None (Async scheduling: overlap the CPU and the GPU explains why these are two calls). On the last PP rank it does five things:

  1. Logits. compute_logits runs only on hidden_states[logits_indices] (vllm/v1/worker/gpu/model_runner.py:1637-1638).
  2. Grammar bitmask. If there is a grammar output, the structured-output worker copies the bitmask to the GPU on a side stream and masks the logits in place (vllm/v1/worker/gpu/model_runner.py:1643-1657). The mask is int32 [rows, ceil(vocab_size / 32)] (vllm/v1/worker/gpu/structured_outputs.py:63-65). Token masks are explained in Structured outputs.
  3. Sampler. Without drafts, the Sampler draws the tokens; with drafts, the RejectionSampler verifies them (vllm/v1/worker/gpu/model_runner.py:1659-1677).
  4. AsyncOutput. The runner starts the copy to the CPU before it updates its own state, so the copy can start early (vllm/v1/worker/gpu/model_runner.py:2209-2218, vllm/v1/worker/gpu/model_runner.py:2244-2255). postprocess_sampled then writes the new token into last_sampled_tokens on the GPU.
  5. Speculator. With speculative decoding, speculator.propose(...) drafts the next tokens on the GPU, and the runner stores them in req_states.draft_tokens (vllm/v1/worker/gpu/model_runner.py:2257-2289). The method is on Speculative decoding in vLLM serving.

Step 9: AsyncOutput and get_output#

vllm/v1/worker/gpu/async_utils.py:133-140GitHub ↗
        # Blocking (sleep) event to avoid busy-polling the CUDA driver lock.
        self.copy_event = torch.cuda.Event(blocking=True)
        self._has_fault: torch.Tensor | None = None

        with stream(copy_stream, main_stream):
            copy_stream.wait_stream(main_stream)

            self.sampled_token_ids = async_copy_to_np(sampler_output.sampled_token_ids)

The copy runs on output_copy_stream, a second CUDA stream of the runner (vllm/v1/worker/gpu/model_runner.py:235). The copy stream first waits for the main stream, then copies into CPU memory, then records copy_event (vllm/v1/worker/gpu/async_utils.py:171). get_output() is the only place that waits:

vllm/v1/worker/gpu/async_utils.py:173-184GitHub ↗
    def get_output(self) -> ModelRunnerOutput:
        self.copy_event.synchronize()

        # NOTE(woosuk): The following code is to ensure compatibility with
        # the existing model runner.
        # Going forward, we should keep the data structures as NumPy arrays
        # rather than Python lists.
        sampled_token_ids: list[list[int]] = self.sampled_token_ids.tolist()
        num_sampled_tokens: list[int] = self.num_sampled_tokens_np.tolist()
        for token_ids, num_tokens in zip(sampled_token_ids, num_sampled_tokens):
            del token_ids[num_tokens:]
        self.model_runner_output.sampled_token_ids = sampled_token_ids

Who calls it depends on the executor:

Step 10: model states and the pooling runner#

Model-specific input work lives in a ModelState, chosen once after the model loads (vllm/v1/worker/gpu/model_runner.py:438-440):

vllm/v1/worker/gpu/model_states/__init__.py:28-59GitHub ↗
def resolve_model_state_cls(
    vllm_config: VllmConfig, model: nn.Module
) -> type[ModelState]:
    # Let the model provide its own ModelState if it defines one.
    if hasattr(model, "get_model_state_cls"):
        return model.get_model_state_cls()

    # Cross-attention encoder-decoder models (Whisper, CohereASR, NemotronParse, ...)
    if any(isinstance(m, CrossAttention) for m in model.modules()):
        from vllm.v1.worker.gpu.model_states.encoder_decoder import (
            EncoderDecoderModelState,
        )

        return EncoderDecoderModelState

    # Encoder-only attention is non-causal and needs no KV cache.
    if any(
        layer.attn_type == AttentionType.ENCODER_ONLY
        for layer in get_layers_from_vllm_config(vllm_config, Attention).values()
    ):
        from vllm.v1.worker.gpu.model_states.encoder_only import EncoderOnlyModelState

        return EncoderOnlyModelState

    if vllm_config.model_config.is_hybrid or vllm_config.model_config.is_attention_free:
        from vllm.v1.worker.gpu.model_states.mamba_hybrid import MambaHybridModelState

        return MambaHybridModelState

    from vllm.v1.worker.gpu.model_states.default import DefaultModelState

    return DefaultModelState

A model can bring its own class with get_model_state_cls(). Otherwise cross-attention models get EncoderDecoderModelState, encoder-only attention gets EncoderOnlyModelState, hybrid and attention-free models get MambaHybridModelState, and everything else, Llama included, gets DefaultModelState. The default state adds M-RoPE positions when the model needs them (vllm/v1/worker/gpu/model_states/default.py:144-157) and builds the attention metadata (vllm/v1/worker/gpu/model_states/default.py:167-242).

A pooling model gets a PoolingRunner on the last PP rank (vllm/v1/worker/gpu/model_runner.py:534-535); its pool() replaces sample_tokens (vllm/v1/worker/gpu/model_runner.py:2316-2358).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

Deep dive

The persistent tensors: shapes, types, where they live

All tensors have max_num_reqs rows, indexed by slot. Values for vllm serve meta-llama/Llama-3.1-8B-Instruct on one H100: max_num_seqs = 1024 (The V1 scheduler: one token budget, no phases) and max_model_len = 131072 (from the model's config.json, inferred).

TensorType, shapeMemoryLine
all_token_idsint32 [1024, 131072] = 512 MiBUVA (pinned CPU memory that kernels read), unless n-gram drafting is on (vllm/v1/worker/gpu/model_runner.py:281-283)vllm/v1/worker/gpu/states.py:37-42
prompt_len, prefill_lenint32 [1024]CPU source of truth, copied to UVA buffersvllm/v1/worker/gpu/states.py:52-53
total_len, num_computed_tokensint32 [1024]GPU, with staged writesvllm/v1/worker/gpu/states.py:55-63
num_computed_tokens_npint32 [1024]CPU; an upper bound of the GPU valuevllm/v1/worker/gpu/states.py:64-65
last_sampled_tokensint64 [1024, 1]GPUvllm/v1/worker/gpu/states.py:67-70
draft_tokensint64 [1024, num_speculative_steps]GPUvllm/v1/worker/gpu/states.py:75-81
block table, per KV cache groupint32 [1024, max_num_blocks]GPU, with staged writesvllm/v1/worker/gpu/block_table.py:66-72
slot mappingsint64 [num_groups, max_num_batched_tokens]GPU, rewritten every stepvllm/v1/worker/gpu/block_table.py:91-96

The comment on all_token_ids warns that it "can be extremely large (e.g., several GBs)" and gives this as the reason for UVA (vllm/v1/worker/gpu/states.py:32-36). A StagedWriteTensor keeps its base tensor on the GPU (or UVA) and collects writes on the CPU. apply_write() packs them and applies them with one kernel (vllm/v1/worker/gpu/buffer_utils.py:132-175, docs/design/model_runner_v2.md:104-130).

The per-step input buffers are separate: input_ids int32, positions int64 and is_padding bool, each [max_num_batched_tokens], plus query_start_loc [max_num_reqs + 1] and seq_lens [max_num_reqs] (vllm/v1/worker/gpu/input_batch.py:19-35). CUDA graphs need these fixed addresses.

Deep dive

Worked example: three requests in one step

Block size 16, no speculative decoding, one KV cache group. Three requests are running:

  • R1 arrived first, so it has slot 1023. It decodes; 40 tokens are computed.
  • R2 has slot 1022. It decodes; 17 tokens are computed.
  • R3 is new, with a 6-token prompt. It gets slot 1021.

The scheduler sends num_scheduled_tokens = {"R3": 6, "R1": 1, "R2": 1}. (Derived from the rules above; the dict order is only an example.)

  1. Sort. Keys: R1 and R2 are (True, False, 1), R3 is (True, True, 6). sorted keeps the dict order for equal keys, so the batch is [R1, R2, R3].
  2. idx_mapping. [1023, 1022, 1021], int64 on the GPU.
  3. query_start_loc. cumsum of [1, 1, 6]: [0, 1, 2, 8]. Entries 4 to 1024 are 8.
  4. positions. [40, 17, 0, 1, 2, 3, 4, 5]; seq_lens [41, 18, 6].
  5. input_ids. Row 0 is last_sampled_tokens[1023]; row 1 is last_sampled_tokens[1022]; rows 2-7 are all_token_ids[1021, 0:6].
  6. logits_indices. The last row of each request: [0, 1, 7]. R3 samples its first token in this step, because its whole prompt fits.
  7. slot mapping. R1's token at position 40 is in block index 2 of its table, offset 8, so its slot is table[1023][2] × 16 + 8. R3's tokens go to table[1021][0] × 16 + 0 .. 5.
  8. CUDA graph mode. The batch has a prefill, so it is not a uniform decode. With the default mode it runs as a PIECEWISE graph of 8 tokens, which is a capture size.

In the next step, all three decode: 3 tokens, one per request. That batch is a uniform decode, so it replays a FULL graph padded to 4 tokens and 4 requests. The fourth request is padding with seq_len = 0 (vllm/v1/worker/gpu/input_batch.py:360-366).

Deep dive

execute_model, end to end

  1. Apply the plan: finish, free, add, update, apply block-table writes (lines 1755-1760).
  2. gather_batch_req_state: sort, idx_mapping_np, prefill flags, uniform decode count (vllm/v1/worker/gpu/model_runner.py:1777-1780).
  3. Count active LoRA adapters, and force eager mode for an encoder-decoder step with encoder inputs (vllm/v1/worker/gpu/model_runner.py:1791-1804).
  4. dispatch_cg_and_sync_dp: pick the descriptor; with DP > 1, agree on one padded size across ranks (vllm/v1/worker/gpu/model_runner.py:1806-1819, vllm/v1/worker/gpu/dp_utils.py:201-255).
  5. prepare_inputs, prepare_attn, model_state.preprocess_state, LoRA activation (vllm/v1/worker/gpu/model_runner.py:1831-1859).
  6. Slot mappings by layer and attention metadata (vllm/v1/worker/gpu/model_runner.py:1924-1951).
  7. Inputs embeddings for multimodal or prompt-embeds models, on the first PP rank only (vllm/v1/worker/gpu/model_runner.py:1953-1984).
  8. Build model_inputs; on a later PP stage, copy the received intermediate tensors into the persistent buffer (vllm/v1/worker/gpu/model_runner.py:1986-2012).
  9. Run the model (FULL, PIECEWISE or eager), store ExecuteModelState, return None, or IntermediateTensors on a non-last PP rank (vllm/v1/worker/gpu/model_runner.py:2103-2110).

A dummy run takes the same function with dummy_run=True: the profile run, warm-up and empty DP steps all use it. Only CUDA graph capture has its own path (docs/design/model_runner_v2.md:171-186).

Deep dive

Speculative decoding and structured outputs inside the runner

Drafts. init_speculator picks the speculator class from the method: EAGLE, MTP, DFlash, DSpark, draft model and others (vllm/v1/worker/gpu/spec_decode/__init__.py:13-80). Only the last PP rank builds one (vllm/v1/worker/gpu/model_runner.py:300-304). The drafts never go to the CPU before the next step. combine_sampled_and_draft_tokens writes them from req_states.draft_tokens into input_ids (line 1500 above). With k drafts, a decode has 1 + k query tokens, which is decode_query_len (lines 442-445 above).

Bitmask. Each structured-output request owns decode_query_len rows of the bitmask. apply_grammar_bitmask maps them to logits rows and launches one masking kernel; rows beyond the device cu_num_logits are skipped (vllm/v1/worker/gpu/structured_outputs.py:70-128). With drafts, grammar_invalid_drafts marks draft positions that the grammar cannot accept, and the rejection sampler gets that mask (vllm/v1/worker/gpu/model_runner.py:1653-1657).

Deep dive

Settings and environment variables

NameDefaultEffect
VLLM_USE_V2_MODEL_RUNNERunset (vllm/envs.py:2147-2149)Unset: MRV2 when Triton is present and no unsupported feature is set. 0/1 force a runner, except that HiSparse and watermarking need MRV2 (vllm/config/vllm.py:713-734).
--max-num-seqsby GPU memory (The V1 scheduler: one token budget, no phases)The number of request slots, max_num_reqs (vllm/v1/worker/gpu/model_runner.py:232).
--max-num-batched-tokensby GPU memoryThe size of the per-step input buffers (vllm/v1/worker/gpu/model_runner.py:327-331).
VLLM_MOE_SKIP_PADDING1 (vllm/envs.py:1640)Marks the padding rows in is_padding (vllm/v1/worker/gpu/model_runner.py:1385-1388), so the MoE routers can skip them (vllm/model_executor/layers/fused_moe/router/fused_topk_router.py:20).

What can go wrong#

Why it matters. Every token of every request passes through this file. Its design explains two numbers you tune. max_num_seqs sets the slots and the size of the persistent tensors. max_num_batched_tokens sets the input buffers and the largest step. If you add code here, keep the rule of the design document: queue GPU work, never wait for it.

Check yourself#

  1. 1
    Which slot does the first request get when max_num_seqs = 1024, and why?
    easy
    Show answer

    Slot 1023. free_indices is list(range(1024)) and add_request takes free_indices.pop(), the last element (vllm/v1/worker/gpu/states.py:30, vllm/v1/worker/gpu/states.py:100).

  2. 2
    A batch has a 300-token prefill chunk, a 1-token decode and a 12-token chunk. In which order does MRV2 put them, and why?
    medium
    Show answer

    Decode, then the 12-token chunk, then the 300-token chunk. The sort key is (no drafts, token count ≠ decode_query_len, token count), so the decode comes first and the others follow by size (vllm/v1/worker/gpu/model_runner.py:2477-2483). Attention backends expect the decode-like rows at the front.

  3. 3
    Where does a decode's input token come from, if the CPU has not received the previous token yet?
    medium
    Show answer

    From req_states.last_sampled_tokens on the GPU. postprocess_sampled writes it in sample_tokens (vllm/v1/worker/gpu/model_runner.py:1702-1726), and combine_sampled_and_draft_tokens copies it into input_ids in the next prepare_inputs (vllm/v1/worker/gpu/model_runner.py:1493-1504).

  4. 4
    With TP = 2, which thread calls AsyncOutput.get_output(), and on which workers?
    hard
    Show answer

    Only on the output rank, because only that rank sends a reply. With async scheduling, its WorkerAsyncOutputCopy thread takes the output from async_output_queue and calls enqueue_output, which calls get_output() (vllm/v1/executor/multiproc_executor.py:1000-1039, vllm/v1/executor/multiproc_executor.py:1070-1071). With TP = 1 and the uni executor, the engine core's main thread calls it through AsyncOutputFuture.result() (vllm/v1/executor/uniproc_executor.py:38-48).

  5. 5
    How much host memory does all_token_ids take for max_num_seqs = 1024 and max_model_len = 131072, and why is it not on the GPU?
    hard
    Show answer

    1024 × 131072 × 4 bytes = 512 MiB of int32. It is a UVA tensor: pinned CPU memory that GPU kernels can read. The comment gives the reason: the tensor "can be extremely large". Only n-gram drafting keeps it on the GPU (vllm/v1/worker/gpu/states.py:32-42, vllm/v1/worker/gpu/model_runner.py:281-283).

  • "v1/worker/gpu/model_runner.py: Model Runner V2"vllm/v1/worker/gpu/model_runner.py holds GPUModelRunner, the default model runner (MRV2). It loads the model, sets up the KV cache, attention backends and CUDA graphs, and per step turns a SchedulerOutput into GPU inputs, runs the forward pass and samples into an AsyncOutput.
  • The forward context: how layers find per-step dataAttention layers register themselves by name when the model is built, get their KV cache tensor bound at start-up, and read the per-step metadata from a global forward context that the model runner sets around each forward pass. The attention custom ops take only the layer name, so torch.compile sees them as opaque.
  • Attention backends: backend, metadata builder and implementationAn attention backend is three classes. The backend class describes what it supports, the metadata builder turns the common batch description of a KV cache group into backend metadata once per step, and the implementation writes the KV cache and runs the kernel for one layer. The builders also report how much CUDA graph capture they allow.
  • torch.compile and CUDA graphs in vLLMvLLM compiles the inner model once with torch.compile and its own Inductor backend, splits the graph at the attention ops, caches the result on disk, and captures CUDA graphs at start-up. Each step, Model Runner V2 replays a FULL graph for uniform decode batches, PIECEWISE graphs for other batches up to the largest capture size, and runs eagerly above it.
  • Async scheduling: overlap the CPU and the GPUWith async scheduling, the engine core schedules and starts step N+1 before it waits for the token of step N. Placeholders in the scheduler and a GPU-side token buffer in Model Runner V2 make this possible.
  • The KV cache manager: blocks, the block pool and memory sizingIn the engine core, the KVCacheManager hands out KV blocks to the scheduler through a coordinator, one single-type manager per KV cache group and one BlockPool. The workers only receive block ids. The number of blocks comes from a memory profile at start-up.
  • The hybrid KV cache: specs, groups and one manager per layer typeModels that mix full attention with sliding-window, chunked-local or Mamba layers get one KV cache group per layer type. Each group has its own block table and single-type manager, all groups share one block pool, and so every group must use the same page size. Prefix caching finds one hit length that is valid for all groups.
  • Tour: One engine stepFollow one call of step_with_batch_queue in 12 stops, with two requests in the batch (a 1-token decode and an 8191-token prefill chunk): schedule, launch, sample, pop the previous step, update and send.
  • Tour: One forward passFollow one forward pass of Llama-3-8B on one H100 in 14 stops, for a batch of two decodes and one 298-token prompt: Model Runner V2 inputs, the PIECEWISE CUDA graph, one decoder layer with its shapes, FlashAttention, logits and the sampler.