Speculative decoding in vLLM serving

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

Speculative decoding lets a cheap drafter guess the next tokens. The large target model then checks all guesses in one forward pass. A rejection test keeps the output distribution of the target model (Speculative decoding: draft, verify, accept/reject↗). This page shows how vLLM does this inside vllm serve.

You turn it on with one flag, --speculative-config, a JSON object that becomes a SpeculativeConfig. The method names the drafter: an EAGLE head, an MTP layer of the model, a separate draft model, an n-gram lookup, and others. num_speculative_tokens (k) sets how many drafts each request gets per step.

In Model Runner V2 the drafts stay on the GPU until the next step checks them. Only a batch with structured outputs copies them to the CPU for the grammar check (vllm/v1/worker/gpu/spec_decode/utils.py:64-80). The scheduler only reserves room for them, with -1 placeholders. The worker samples, drafts the next k tokens right away on the GPU, and puts them into the next input. The rejection sampler checks them in the next step.

An everyday comparison. Think of a junior writer and an editor. The junior writes the next three words. The editor reads them in one pass, keeps the words up to the first wrong one, and writes one correct word there. If all three are right, the editor adds a fourth word. The comparison stops being true in one place: the editor is not slower per word than the junior. The target model checks four positions for about the price of one decode step, because decode is memory-bound (Prefill and decode↗).

The picture#

Read the panels from top to bottom. Blue and teal are the target forward pass and the check, amber and violet are the GPU work after sampling, emerald is the engine core. The numbers are one example request; the worked example in the deep dive follows the same numbers.

In the code#

Step 1: SpeculativeConfig and the methods#

--speculative-config (short -sc) takes a JSON string. vLLM keeps it as a dict until it builds the engine config, because SpeculativeConfig needs the target model config (vllm/engine/arg_utils.py:1851-1857). Three short flags exist too: --spec-method, --spec-model and --spec-tokens. A short flag and the same key in the JSON raise ValueError (vllm/engine/arg_utils.py:2098-2110).

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-config '{"method": "eagle3",
                         "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B",
                         "num_speculative_tokens": 3}'

The method names are a Literal type:

NgramGPUTypes = Literal["ngram_gpu"]
DFlashModelTypes = Literal["dflash"]
DSparkModelTypes = Literal["dspark"]
EagleModelTypes = Literal[
    "eagle", "eagle3", "extract_hidden_states", MTPModelTypes, DFlashModelTypes
]
SpeculativeMethod = Literal[
    "ngram",
    "draft_model",
    "suffix",
    "custom_class",
    EagleModelTypes,
    NgramGPUTypes,
    DSparkModelTypes,
]
RejectionSampleMethod = Literal["standard", "synthetic", "block", "fly"]
DraftSampleMethod = Literal["greedy", "probabilistic"]

• Draft heads that read the target's hidden states: eagle, eagle3, mtp, dflash and dspark. use_eagle() is true for all five (vllm/config/speculative.py:2013-2017). dflash and dspark always draft in parallel (vllm/config/speculative.py:1505-1506).
• MTP. The 27 model-specific MTP names in MTPModelTypes (for example deepseek_mtp) are deprecated (vllm/config/speculative.py:38-67). vLLM logs a warning and uses mtp (vllm/config/speculative.py:1197-1201). Without model, MTP uses the target checkpoint itself (vllm/config/speculative.py:1204-1215).
• A separate small model: draft_model. It is also the fallback when method is missing and the model name gives no other hint (vllm/config/speculative.py:1186-1195).
• No model at all: ngram and ngram_gpu look up the last tokens in the prompt and the output. Both look-up sizes default to 5 when you set neither (vllm/config/speculative.py:1252-1256).
• Others: suffix (suffix decoding), custom_class (your own proposer class, "experimental") and extract_hidden_states (no drafting; it only exports hidden states) (vllm/config/speculative.py:1286-1293, vllm/config/speculative.py:1298-1304).

With a model but no method, vLLM guesses the method from the model name and architecture. It looks for "eagle-", "eagle3", "dflash", "dspark" or an MTP model_type (vllm/config/speculative.py:1363-1398). num_speculative_tokens has no default. It comes from n_predict in the draft config when that exists (for DSpark, from its block size); otherwise it is required (vllm/config/speculative.py:1508-1542).

Step 2: which methods Model Runner V2 supports#

Model Runner V2 is the default. Some speculative settings switch the whole worker back to Model Runner V1 (The model runner (Model Runner V2)):

        if speculative_config is not None:
            if speculative_config.method in ("suffix", "custom_class"):
                unsupported.append(f"speculative method '{speculative_config.method}'")

            # V2 EagleSpeculator does not support parallel_drafting (for P-Eagle).
            # DFlash and DSpark use parallel drafting natively in V2 via their
            # own speculators.
            if (
                speculative_config.parallel_drafting
                and speculative_config.method not in ("dflash", "dspark")
            ):
                unsupported.append("parallel drafting for EAGLE speculative decoding")

            # The V2 draft-model speculator has no token mapping between the
            # draft and target vocabularies (TLI is TBD in #47172).
            if getattr(speculative_config, "use_heterogeneous_vocab", False):
                unsupported.append("heterogeneous-vocabulary draft models")

So suffix, custom_class, parallel drafting with EAGLE (P-EAGLE) and draft models with a different vocabulary run on the fallback. vLLM logs "Model Runner V2 does not yet support ..." (vllm/config/vllm.py:740-747). The reverse also exists: dspark, adaptive verification and draft LM-head quantization work only on Model Runner V2 (vllm/config/vllm.py:3056-3063).

In Model Runner V2, init_speculator maps the method to a speculator class. ngram and ngram_gpu both get the GPU n-gram speculator (vllm/v1/worker/gpu/spec_decode/__init__.py:81-86, vllm/config/speculative.py:2048-2049). Only the last pipeline-parallel rank builds a speculator (vllm/v1/worker/gpu/model_runner.py:298-308).

Step 3: drafts in the scheduler#

The scheduler treats drafts as normal tokens of the request. A decode with k drafts schedules 1 + k tokens, and they count against the token budget. After scheduling, it keeps only the drafts that fit and moves them into scheduled_spec_decode_tokens (vllm/v1/core/sched/scheduler.py:916-932). The page The V1 scheduler: one token budget, no phases (Step 6) shows the budget rule.

Two extra reservations exist:

• Draft slots in the input budget: rows that the drafter adds to the batch. They are 0 for EAGLE, MTP, n-gram, DFlash and DSpark, 1 for a separate draft model, and more with parallel drafting (vllm/config/speculative.py:1955-1995, vllm/v1/core/sched/scheduler.py:644).
• Lookahead KV slots. The drafter writes KV past the target's query. So allocate_slots gets num_lookahead_tokens: k for EAGLE-type and draft-model methods, k + 1 for DFlash, 0 for n-gram (vllm/config/vllm.py:625-650, vllm/v1/core/sched/scheduler.py:834-838).

With async scheduling (the default), the scheduler plans step N+1 before the drafts of step N exist. It puts [-1] * k as placeholder drafts and adds 1 + k output placeholders (vllm/v1/core/sched/async_scheduler.py:28-50). The real drafts are only on the GPU. Async scheduling: overlap the CPU and the GPU explains the placeholders.

Step 4: verify, then draft again, in one sample_tokens call#

In sample(), the model runner chooses the sampler. Without drafts in the batch the normal sampler runs; with drafts the RejectionSampler runs:

        sampler_output: SamplerOutput | None
        if input_batch.num_reqs == 0:
            # This rank owns no requests this step. It contributes an
            # all-padding block to the gather below.
            sampler_output = None
        elif input_batch.num_draft_tokens == 0 or self.rejection_sampler is None:
            assert self.sampler is not None
            sampler_output = self.sampler(logits, input_batch)
        else:
            # Rejection sampling for spec decoding.
            assert self.rejection_sampler is not None
            assert self.speculator is not None
            sampler_output = self.rejection_sampler(
                logits,
                input_batch,
                # Draft logits are needed for probabilistic rejection sampling.
                self.speculator.draft_logits,
                invalid_drafts,
            )

After sampling, sample_tokens starts the output copy (vllm/v1/worker/gpu/model_runner.py:2209-2218), updates the request state (vllm/v1/worker/gpu/model_runner.py:2249-2255), and then calls the speculator at once:

            with use_workspace_lane(self._draft_workspace_lane):
                draft_tokens = self.speculator.propose(
                    input_batch,
                    attn_metadata,
                    slot_mappings_by_layer,
                    spec_hidden_states,
                    aux_hidden_states,
                    num_sampled,
                    num_rejected,
                    self.req_states.last_sampled_tokens,
                    self.req_states.next_prefill_tokens,
                    self.sampler.sampling_states.temperature.gpu,
                    self.sampler.sampling_states.seeds.gpu,
                    num_speculative_tokens=num_spec_tokens,
                    dp_sync_state=dp_sync_state,
                    mm_inputs=mm_inputs,
                )
            if num_spec_tokens < self.num_speculative_steps:
                draft_tokens[:, num_spec_tokens:] = -1
            self.req_states.draft_tokens[input_batch.idx_mapping] = draft_tokens

1. Line 2271: propose() gets the hidden states, the number of sampled and rejected tokens, and the last sampled token, all on the GPU.
2. Lines 2287-2288: with dynamic speculative decoding the step may use fewer drafts; the unused columns become -1.
3. Line 2289: the drafts go into req_states.draft_tokens. The next prepare_inputs copies them into input_ids (vllm/v1/worker/gpu/model_runner.py:1493-1504).

The EAGLE speculator first runs a "draft prefill" over the same token rows as the target. So it can reuse the target's attention metadata and slot mappings. Then it runs k - 1 decode steps of one token per request (vllm/v1/worker/gpu/spec_decode/target_dependent_ar/speculator.py:204-209, vllm/v1/worker/gpu/spec_decode/target_dependent_ar/speculator.py:281-299, vllm/v1/worker/gpu/spec_decode/target_dependent_ar/speculator.py:331-366).

Step 5: the RejectionSampler#

The RejectionSampler wraps the normal sampler. First it applies the sampling parameters of each request (penalties, temperature, top-k, top-p) to all 1 + k logits rows; then it runs the test (vllm/v1/worker/gpu/spec_decode/rejection_sampler.py:264-293). The test depends on rejection_sample_method:

    rejection_sample_method: RejectionSampleMethod = "standard"
    """The rejection sampling method to use. 'standard' uses probabilistic
    rejection sampling (with or without cached draft logits, controlled by
    draft_sample_method). 'synthetic' accepts draft tokens with a decaying
    probability calibrated to synthetic_acceptance_rate. 'block' uses block
    verification (Sun et al.), which jointly verifies the draft tokens as a
    block instead of one at a time. 'fly' applies FLy's lossy, entropy-gated
    deferred window to the native per-token acceptance decisions."""

The acceptance math. p is the target probability and q the draft probability of the draft token x, both after temperature. u is a uniform random number. The draft is kept when p(x) > u · q(x), that is with probability min(1, p(x) / q(x)). The default draft_sample_method is greedy: the drafter takes the argmax, and q counts as one-hot. So q(x) = 1, and the draft is kept with probability p(x) (vllm/config/speculative.py:608-614, vllm/v1/worker/gpu/spec_decode/rejection_sampler_utils.py:191-194). At the first rejection, vLLM draws a new token from the residual max(p - q, 0). With a one-hot draft, that is the target distribution without the rejected token (vllm/v1/worker/gpu/spec_decode/rejection_sampler_utils.py:937-938, vllm/v1/worker/gpu/spec_decode/rejection_sampler_utils.py:950-951). If all k drafts pass, the last row gives a bonus token.

Step 6: the engine core rolls back rejected drafts#

The model runner already corrected its own GPU counters. num_rejected is the number of logits rows minus the number of sampled tokens, and num_computed_tokens grows by the query length minus that (vllm/v1/worker/gpu/input_batch.py:528-534, vllm/v1/worker/gpu/input_batch.py:612-615). The scheduler does the same on the CPU when the output arrives:

            scheduled_spec_token_ids = (
                scheduler_output.scheduled_spec_decode_tokens.get(req_id)
            )
            if scheduled_spec_token_ids and (
                generated_token_ids or self.num_sampled_tokens_per_step == 0
            ):
                num_draft_tokens = len(scheduled_spec_token_ids)
                num_sampled = self.num_sampled_tokens_per_step
                num_accepted = max(len(generated_token_ids) - num_sampled, 0)
                num_rejected = num_draft_tokens - num_accepted
                # Rejections roll back num_computed_tokens (and, under async
                # scheduling, num_output_placeholders, which covers the spec
                # tokens). A stale rejection count predates the preemption
                # rollback and must not apply.
                if not output_is_stale:
                    if request.num_computed_tokens > 0:
                        request.num_computed_tokens -= num_rejected
                    if request.num_output_placeholders > 0:
                        request.num_output_placeholders -= num_rejected
                spec_decoding_stats = self.make_spec_decoding_stats(
                    spec_decoding_stats,
                    num_draft_tokens=num_draft_tokens,
                    num_accepted_tokens=num_accepted,
                    num_invalid_spec_tokens=scheduler_output.num_invalid_spec_tokens,
                    request_id=req_id,
                )

The KV entries of rejected drafts are not freed. The blocks stay with the request: the KV manager notes that a request can hold more blocks than it needs after a rejection (vllm/v1/core/single_type_kv_cache_manager.py:233-236). The next step writes the correct token over the stale entry, because num_computed_tokens moved back (derived from the counters above). With async scheduling, post_step does not take draft tokens from the workers (vllm/v1/engine/core.py:689-696). The engine core takes them only for the grammar check of a structured-output step (vllm/v1/engine/core.py:792-803).

Step 7: outputs with more than one token#

One step can give a request up to 1 + k new tokens, and they travel in one EngineCoreOutput. This changes the latency numbers:

• ITL. The front end adds one inter-token latency sample per output, not per token (vllm/v1/metrics/stats.py:551-557). With 3 tokens per step, there are a third as many ITL samples, and each is longer.
• TPOT. The decode time is divided by all output tokens minus one, so TPOT goes down (vllm/v1/metrics/stats.py:612-616). See Serving metrics: TTFT, TPOT, ITL, throughput, goodput.
• Streaming. A stream gets the new tokens of a step together, so one SSE chunk can hold several tokens of text.

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

What can go wrong#

• "num_speculative_tokens was provided but without speculative model." You set k for a method that needs a model, without model (vllm/config/speculative.py:1243-1246).
• "Target and draft model should have the same vocabulary size." A draft_model with another tokenizer (vllm/config/speculative.py:1937-1953).
• "The min_p and logit_bias sampling parameters are not yet supported with speculative decoding." Such a request gets a validation error (vllm/sampling_params.py:1192-1197).
• ITL looks worse, TPOT better. This is expected: one output holds several tokens (Step 7).
• Low acceptance with structured outputs. Drafts that the grammar rejects become -1 before verification and count as invalid (Structured outputs: JSON schemas, grammars and token bitmasks in vLLM).
• FLy or synthetic in production. Both change the output distribution; FLy logs "FLy verification is a lossy speculative decoding method" (vllm/config/speculative.py:1928-1931).

Why it matters. Speculative decoding trades extra GPU work per step for fewer steps per answer. It helps most at small batch sizes, where decode leaves the GPU idle. The acceptance metrics tell you whether the trade pays off.