Sampling and logits processors in Model Runner V2

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

Sampling picks the next token from the logits. Temperature, top-k and top-p shape the distribution, and a seed makes the random draw repeatable (Sampling: temperature, top-k, top-p, seeds↗). This page shows how vLLM's sampler does these steps for a whole batch, and where your own code can change the logits.

Each request brings its SamplingParams. When a request enters the batch, the sampler writes its settings into per-slot GPU buffers, one row per request slot. In every step, it works on all logits rows at once. First come the logits processors (logit bias, penalties, bad words, then yours). Then come temperature, min_p, top-k and top-p, and then the draw. A stage that no request in the batch uses is skipped, without a kernel launch.

An everyday comparison. Think of a mail sorting line. Each letter (a logits row) passes the same stations in a fixed order: stamps, weight check, address check, then the bin. Each letter carries a slip that says which stations apply to it. The comparison stops being true in one place. vLLM runs a whole tray through each station at once, and a station turns on when one letter needs it.

The picture#

Read the panels from top to bottom. Teal stages are built in, violet is where custom processors run, and emerald is the token draw. The dashed arrow in panel 3 means "or": one batch takes one path.

In the code#

Step 1: the sampling parameters of a request#

The OpenAI server turns the request into a SamplingParams object. Unset fields take the server default, which comes from the model's generation_config.json with the default --generation-config auto (The OpenAI-compatible API). These fields change the sampler:

A temperature below 1e-5 means greedy: vLLM resets top_p, top_k and min_p and requires n = 1 (vllm/sampling_params.py:615-620, vllm/sampling_params.py:769-773). In the OpenAI API, vllm_xargs becomes extra_args (vllm/entrypoints/openai/chat_completion/protocol.py:770).

Step 2: per-slot state, written once per request#

The sampler is built in load_model, only on the last pipeline rank and only for generative models (vllm/v1/worker/gpu/model_runner.py:463-467). Its constructor fixes the list of processors:

        self.sampling_states = SamplingStates(max_num_reqs, vocab_size)

        lp_req_state = LogitsProcRequestState.from_request_state(req_states)
        self.penalties_state = PenaltiesState(vllm_config, lp_req_state)
        logit_bias_state = LogitBiasState(vllm_config, lp_req_state)
        bad_words_state = BadWordsState(vllm_config, lp_req_state)

        # List order is pipeline order: bias adds, penalties scale, so the
        # two do not commute.
        self.logits_processors: list[LogitsProcessor] = [
            logit_bias_state,
            self.penalties_state,
            bad_words_state,
            *custom_logits_processors,
        ]

        self.logprob_token_ids_state = LogprobTokenIdsState(max_num_reqs, device)
        self.thinking_budget_state = ThinkingBudgetState(
            req_states, vllm_config.reasoning_config
        )
        self.trace_replay_state = (
            TraceReplayState(req_states) if enable_trace_replay else None
        )
        self.needs_logits_processing = np.zeros(max_num_reqs, dtype=bool)
        self.num_speculative_tokens = num_speculative_tokens
        self.return_sampling_mask = return_sampling_mask
        self.use_flashinfer = (
            not return_sampling_mask and flashinfer_sampler_supported()
        )

When the model runner adds a new request, the sampler calls add_request(slot, params) on every stage. Each stage returns whether it changes the logits of this request; the OR of all answers is stored per slot (vllm/v1/worker/gpu/sample/sampler.py:96-107). The sampling state itself returns true only for a non-zero temperature together with a temperature other than 1, or a set min_p, top-k or top-p:

    def add_request(self, req_idx: int, sampling_params: SamplingParams) -> bool:
        temperature = sampling_params.temperature
        top_k = sampling_params.top_k
        top_p = sampling_params.top_p
        min_p = sampling_params.min_p

        self.temperature.np[req_idx] = temperature
        self.top_p.np[req_idx] = top_p
        if top_k <= 0 or top_k > self.vocab_size:
            top_k = self.vocab_size
        self.top_k.np[req_idx] = top_k
        self.min_p.np[req_idx] = min_p

        seed = sampling_params.seed
        self.seeds_set[req_idx] = seed is not None
        if seed is None:
            seed = np.random.randint(_NP_INT64_MIN, _NP_INT64_MAX)
        self.seeds.np[req_idx] = seed

        num_logprobs = sampling_params.logprobs
        if num_logprobs is None:
            num_logprobs = NO_LOGPROBS
        elif num_logprobs == -1:
            num_logprobs = self.vocab_size
        self.num_logprobs[req_idx] = num_logprobs

        return temperature != 0.0 and (
            temperature != 1.0
            or min_p != 0.0
            or top_k != self.vocab_size
            or top_p != 1.0
        )

The writes are staged on the CPU. The model runner calls sampler.apply_staged_writes() once per step, which copies them to the GPU buffers (vllm/v1/worker/gpu/model_runner.py:1251-1255, vllm/v1/worker/gpu/sample/sampler.py:109-116).

Step 3: the processing chain#

In each step the model runner computes the logits for the sampled rows and applies the grammar bitmask (Structured outputs: JSON schemas, grammars and token bitmasks in vLLM). Then the sampler runs apply_sampling_params:

        if not np.any(self.needs_logits_processing[idx_mapping_np]):
            return logits

        # Copy logits to a new FP32 tensor.
        logits = torch.empty_like(logits, dtype=torch.float32).copy_(logits)

        ctx = LogitsContext(
            expanded_idx_mapping=expanded_idx_mapping,
            idx_mapping=idx_mapping,
            idx_mapping_np=idx_mapping_np,
            expanded_local_pos=expanded_local_pos,
            input_ids=input_ids,
            pos=pos,
            seq_lens_upper_bound_np=seq_lens_upper_bound_np,
        )

        # Apply logits processors (native + any custom).
        for processor in self.logits_processors:
            logits = processor.apply(logits, ctx)

        # Forcing runs last so no stage can overwrite the forced end marker
        # or weaken it by scaling.
        self.thinking_budget_state.apply(logits, ctx)

        # Apply temperature in place.
        self.sampling_states.apply_temperature(
            logits, expanded_idx_mapping, idx_mapping_np
        )

        # Apply min_p in place.
        self.sampling_states.apply_min_p(logits, expanded_idx_mapping, idx_mapping_np)

        if skip_top_k_top_p:
            return logits

        # Apply top_k and/or top_p. This might or might not return a new tensor.
        return self.sampling_states.apply_top_k_top_p(
            logits, expanded_idx_mapping, idx_mapping_np
        )

1. Lines 238-239: if no request in the batch needs processing, the raw logits go straight to the draw. There is no copy.
2. Line 242: else the logits are copied into a new FP32 tensor, [rows, vocab].
3. Lines 255-256: the processors run in list order. LogitBiasState does allowed token ids, logit_bias and min_tokens in one kernel. PenaltiesState does the three penalties. BadWordsState masks the last token of a bad word when the earlier ones match. Then come the custom processors.
4. Line 260: thinking-budget forcing runs after all processors, so no processor can overwrite it (the code comment at lines 258-259).
5. Lines 263-268: temperature and min_p, in place. Each skips its kernel when no request in the batch uses it (vllm/v1/worker/gpu/sample/states.py:87-103).

The order matters. The code comment says that bias adds and penalties scale, "so the two do not commute" (vllm/v1/worker/gpu/sample/sampler.py:69-70).

Step 4: the draw - fused FlashInfer or Gumbel#

        top_k, top_p, has_topp_only_rows = self.sampling_states.get_top_k_top_p(
            expanded_idx_mapping, idx_mapping_np, logits.shape[-1]
        )
        # Don't use a fused sampler if no requests use top_k/top_p, if there are
        # any greedy requests or per-request seeds, or if post-processed
        # logprobs need to be returned for any requests.
        fused_sampler_eligible = not (
            (top_k is None and top_p is None)
            or (return_logprobs and self.logprobs_mode in PROCESSED_LOGPROBS_MODES)
            or self.sampling_states.any_greedy(idx_mapping_np)
            or self.sampling_states.any_explicit_seed(idx_mapping_np)
        )
        use_fused_sampler = (
            self.use_flashinfer or self.use_xpu_sampler
        ) and fused_sampler_eligible

        return self._sample_random(
            processed_logits,
            expanded_idx_mapping,
            idx_mapping_np,
            pos,
            top_k,
            top_p,
            has_topp_only_rows,
            use_fused_sampler,
        )

    def _sample_random(
        self,
        processed_logits: torch.Tensor,
        expanded_idx_mapping: torch.Tensor,
        idx_mapping_np: np.ndarray,
        pos: torch.Tensor,
        top_k: torch.Tensor | None,
        top_p: torch.Tensor | None,
        has_topp_only_rows: bool,
        use_fused_sampler: bool,
    ) -> tuple[torch.Tensor, torch.Tensor]:
        if use_fused_sampler:
            if self.use_flashinfer:
                sampled = flashinfer_sample(processed_logits, top_k, top_p).to(
                    torch.int64
                )
            else:  # Use XPU sampler
                sampled, _ = xpu_sample(processed_logits, top_k, top_p)
        else:
            processed_logits = apply_top_k_top_p(
                processed_logits, top_k, top_p, has_topp_only_rows=has_topp_only_rows
            )
            sampled = gumbel_sample(
                processed_logits,
                expanded_idx_mapping,
                self.sampling_states.temperature.gpu,
                self.sampling_states.seeds.gpu,
                pos,
                apply_temperature=False,
                is_drafting=False,
                use_fp64=self.use_fp64_gumbel,
            )

• Fused path (lines 307-315, 339-343). It runs only when the batch uses top-k or top-p, and no request is greedy, has a seed, or asks for processed logprobs. The sampling mask must be off, which is the default. FlashInfer's sampler must also support the GPU: CUDA, a supported compute capability, more than 16 SMs, and a working JIT build. VLLM_USE_FLASHINFER_SAMPLER must be on, the default (vllm/v1/worker/gpu/sample/sampler.py:88-90, vllm/v1/sample/ops/topk_topp_sampler.py:102-150, vllm/envs.py:886-890). The Tour: One forward pass (stop 14) walks through this for a default request.
• Fallback (lines 347-359). vLLM masks top-k and top-p itself and runs the Gumbel sampler. Gumbel-max adds noise to logits / T and takes the argmax; at temperature 0 it is a plain argmax (vllm/v1/worker/gpu/sample/gumbel.py:182-187).

The conditions are for the whole batch. One greedy or seeded request sends every request of that step to the fallback path.

Step 5: seeds without generators#

Model Runner V2 keeps no torch.Generator per request. The Gumbel noise is a hash of the request's seed, the token position and the token id (vllm/v1/worker/gpu/sample/gumbel.py:197-209). A request without a seed gets a random 64-bit seed when it enters the batch (vllm/v1/worker/gpu/sample/states.py:54-58). So the noise of a seeded request does not depend on the other rows of the batch (derived from the hash inputs above). Its logits must still be the same; see Determinism and batch invariance↗.

Step 6: custom logits processors#

A custom processor subclasses the Model Runner V2 LogitsProcessor:

class LogitsProcessor(ABC):
    """Custom logits processor for Model Runner V2.

    Per-request state is keyed by the request slot index; slots are recycled
    through a free list, so per-slot state must be fully (re)initialized in
    ``add_request()``.

    ``apply()`` runs after the built-in bias, penalty, bad-words and grammar
    stages and before temperature, min_p and top-k/top-p, so it sees unscaled
    logits and must not re-inflate grammar-masked tokens. Thinking-budget
    forcing runs after ``apply()`` and wins over its edits for requests whose
    budget is exhausted.

    State that is constant for a request belongs in ``__init__()`` or
    ``add_request()``; ``apply()`` receives only what changes per step.
    """

The methods are validate_params (class method, at admission), add_request(req_idx, params), apply_staged_writes() and apply(logits, ctx) (vllm/v1/worker/gpu/sample/logits_processor/interface.py:118-158). ctx is a LogitsContext: the slot of each row, the positions, the input ids and a CPU bound of the sequence lengths (vllm/v1/worker/gpu/sample/logits_processor/interface.py:63-89). Rows are reordered every step, so a processor must find its rows through ctx.expanded_idx_mapping.

There are two sources, and the loader puts entry-point plugins first (vllm/v1/worker/gpu/sample/logits_processor/loader.py:136-146):

• the entry-point group vllm.logits_processors, which VLLM_PLUGINS does not filter;
• --logits-processors (ModelConfig.logits_processors), with "module:Class" strings (vllm/config/model.py:407).

Hooks in the engine core and the workers (Step 3) covers loading, errors and the front-end check. The repository has a full example, a DRY repetition penalty that reads extra_args (examples/features/logits_processor/dry.py:659-688).

The Model Runner V2 interface has no "argmax invariant" flag. The legacy V1 interface has is_argmax_invariant, and a BatchUpdate object with the added, removed and moved rows of the batch. Model Runner V2 rejects such classes (vllm/v1/sample/logits_processor/interface.py:37-40, vllm/v1/sample/logits_processor/interface.py:86-87, vllm/v1/worker/gpu/sample/logits_processor/loader.py:67-72).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

{"model": "meta-llama/Llama-3.1-8B-Instruct",
 "messages": [{"role": "user", "content": "Name three rivers."}],
 "temperature": 0.6, "top_p": 0.9, "seed": 7,
 "repetition_penalty": 1.1}

What can go wrong#

• The model's defaults change your sampling. A request without temperature uses the value from generation_config.json, not 1.0. Use --generation-config vllm for vLLM's neutral defaults (The OpenAI-compatible API).
• One seeded request slows the batch. A seed, a greedy request or a processed-logprobs request makes the whole step skip the fused sampler (Step 4).
• A V1 logits processor plugin stops the start-up. Model Runner V2 raises ValueError at model load (vllm/v1/worker/gpu/sample/logits_processor/loader.py:67-72).
• --logits-processors on a pooling model. It raises ValueError. Entry-point processors are skipped for pooling models, with only a debug log (vllm/v1/worker/gpu/sample/logits_processor/loader.py:172-179).
• Too many ids. More than 1,024 allowed ids or bias entries, or more than 128 stop tokens with min_tokens, raise ValueError in the worker (vllm/v1/worker/gpu/sample/logit_bias.py:63-104).

Why it matters. The sampler runs in every step for every request. Know which stage runs and which settings force the slower path. Then you can choose good server defaults, and write processors that cost nothing for other requests.