Tour: One engine step
Code version: vLLM @ 3e44c0d · 2026-10-22
In one sentence
We follow one call of step_with_batch_queue: it plans and starts step k for two requests, then finishes step k-1 and sends its token.
The idea#
Tour 1 (Tour: Life of a chat request) follows one request through the whole server. This tour stays in one place, the engine core process, and looks at one engine step in detail. A real server has many requests in each step, so we take two that need different work:
- Request B is a chat answer in progress. It decodes: it needs 1 new token per step.
- Request A has a long prompt of 20,000 tokens. The prompt does not fit into one step, so the scheduler runs it in chunks (chunked prefill).
With async scheduling, one call of step_with_batch_queue does two things. It plans and starts step k, and then it finishes step k-1, which is already on the GPU. So the token that this call sends belongs to the step before. The page Async scheduling: overlap the CPU and the GPU explains why.
An everyday comparison. Think of a bakery with one oven. The baker puts the next tray into the queue of the oven before the current tray is done. Then the baker takes out the finished tray and packs it. One tray holds rolls (one each, the decodes) and a big part of a cake (the prefill chunk). The comparison stops in one place. The cake leaves the oven unfinished and goes back in the next round. Only a finished cake is packed. A prefill chunk gives no token.
The example#
vllm serve meta-llama/Llama-3.1-8B-Instruct # one H100 80 GiB GPU, default settings- One GPU, TP = 1, DP = 1: the uni executor runs the worker in the engine core process (
vllm/config/parallel.py:1099-1100). - Token budget:
max_num_batched_tokens = 8192andmax_num_seqs = 1024, because the GPU has 70 GiB or more and is not an A100 (vllm/engine/arg_utils.py:2960-2993). An A100 would get 2048 and 256. - Async scheduling is on, and the batch queue has 2 slots (
vllm/config/vllm.py:1664-1698,vllm/config/vllm.py:586-597). - Step k-1 is already on the GPU. In it, B decoded its 41st output token, and A ran its first chunk: prompt positions 0 to 8190.
The picture#
| Stop | What happens | File and function |
|---|---|---|
| 1 | The busy loop calls the step function | v1/engine/core.py: _process_engine_step |
| 2 | The scheduler shares the token budget | v1/core/sched/scheduler.py: schedule |
| 3 | The SchedulerOutput and the placeholders | scheduler.py; async_scheduler.py: _update_after_schedule |
| 4 | The executor starts execute_model | v1/executor/uniproc_executor.py; v1/worker/gpu_worker.py |
| 5 | Model Runner V2 updates its states and picks a CUDA graph mode | v1/worker/gpu/model_runner.py: execute_model |
| 6 | Inputs and attention metadata | model_runner.py: prepare_inputs, prepare_attn |
| 7 | The forward pass is queued; execute_model returns None | model_runner.py: execute_model |
| 8 | The grammar bitmask (none here) | core.py; scheduler.py: get_grammar_bitmask |
| 9 | sample_tokens: logits, sampler, AsyncOutput | model_runner.py: sample_tokens |
| 10 | The batch queue is full: pop step k-1 and wait | core.py: step_with_batch_queue; gpu/async_utils.py |
| 11 | update_from_output for step k-1 | scheduler.py: update_from_output |
| 12 | EngineCoreOutputs go to the output thread | core.py: process_output_sockets |
All paths are under vllm/. All stops run in the engine core process.
1The busy loop calls step_with_batch_queue#
We are in the main thread of the engine core process, in EngineCoreProc._process_engine_step. The input queue is empty, and the scheduler has two running requests, so the busy loop does not block (vllm/v1/engine/core.py:1596-1602). It calls the step function:
def _process_engine_step(self) -> bool:
"""Called only when there are unfinished local requests."""
get_watch_dog().feed()
# Step the engine core.
outputs, model_executed = self.step_fn()
# Put EngineCoreOutputs into the output queue.
for output in outputs.items() if outputs else ():
self.output_queue.put_nowait(output)
# Post-step hook.
self.post_step(model_executed)step_fn is step_with_batch_queue, because the batch queue exists (vllm/v1/engine/core.py:227-229). Right now the queue holds one entry: step k-1.
batch_queue = self.batch_queue
assert batch_queue is not None
# Try to schedule a new batch if the batch queue is not full, but
# the scheduler may return an empty batch if all requests are scheduled.
# Note that this is not blocking.
assert len(batch_queue) < self.batch_queue_size
model_executed = False
deferred_scheduler_output = None
if self.scheduler.has_requests():
scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())What the data looks like now:
batch_queue = deque([(future_sample[k-1], scheduler_output[k-1], future_exec[k-1])], maxlen=2)
scheduler.running = [B, A] # B first: it was running before A arrived2Scheduler.schedule(): sharing 8192 tokens#
We are in vllm/v1/core/sched/scheduler.py. The scheduler has no prefill phase and no decode phase. It gives every running request the tokens it still needs, until the budget is used (see The V1 scheduler: one token budget, no phases). The running pass computes the need of each request:
num_new_tokens = (
request.num_tokens_with_spec
+ request.num_output_placeholders
- request.num_computed_tokens
)
if 0 < long_prefill_token_threshold < num_new_tokens:
num_new_tokens = long_prefill_token_threshold
num_new_tokens = min(
num_new_tokens, token_budget, input_budget - draft_slots
)- B. B knows 540 tokens (500 prompt + 40 output). Token 41 is in flight, so B has 1 output placeholder, and
num_computed_tokens = 540. So540 + 1 - 540 = 1. The budget drops from 8192 to 8191. - A. A has 20,000 prompt tokens, no placeholder, and 8191 computed tokens. Its need is
20000 - 8191 = 11809, but line 766 caps it at the budget: 8191. The budget is now 0.
The order matters: the running pass goes through self.running from the front (vllm/v1/core/sched/scheduler.py:713-716). B was first, so the decode gets its token before the prefill takes the rest. long_prefill_token_threshold could cap A further, but its default is 0, which means no cap (vllm/config/scheduler.py:79-84). The waiting pass has no budget left, so no new request starts in this step.
Then allocate_slots reserves KV cache slots for the new tokens (vllm/v1/core/sched/scheduler.py:831-838). With blocks of 16 tokens, A needs ceil(16382 / 16) = 1024 blocks; it had 512, so it gets 512 new blocks. B needs ceil(541 / 16) = 34 blocks, the same as before. (Derived.) The KV cache and its blocks are explained in The KV cache.
3The SchedulerOutput, and the placeholder for B#
Both requests ran before, so they travel as CachedRequestData, which carries only the changes (vllm/v1/core/sched/output.py:137-152). The scheduler builds it before it advances the counters (vllm/v1/core/sched/scheduler.py:1527-1534), then builds the SchedulerOutput (vllm/v1/core/sched/scheduler.py:1588-1609), and only then calls _update_after_schedule (vllm/v1/core/sched/scheduler.py:1641-1643). That method advances num_computed_tokens and decides whether each request is still in a prefill chunk:
num_scheduled_tokens = scheduler_output.num_scheduled_tokens
for req_id, num_scheduled_token in num_scheduled_tokens.items():
request = self.requests[req_id]
request.num_computed_tokens += num_scheduled_token
request.num_in_flight_tokens += num_scheduled_token
if self.defer_block_free:
# Record the in-flight step, to fence deferred block freeing.
request.last_sched_seq = self.sched_step_seq
request.is_prefill_chunk = request.num_computed_tokens < (
request.num_tokens + request.num_output_placeholders
)For A: 8191 + 8191 = 16382 < 20000, so A is still a prefill chunk. For B: 541 < 540 + 1 is false, so B is not. AsyncScheduler then adds placeholders only for B:
for req_id in scheduler_output.num_scheduled_tokens:
request = self.requests[req_id]
if request.is_prefill_chunk:
continue
scheduler_output.pending_structured_output_tokens |= (
request.use_structured_output and request.num_output_placeholders > 0
)
# The request will generate num_sampled_tokens_per_step new tokens
# plus num_spec_tokens in this scheduling step. Diffusion has no AR
# bonus token (num_sampled_tokens_per_step == 0) — only the canvas
# (spec) tokens.
cur_num_spec_tokens = len(spec_decode_tokens.get(req_id, ()))
request.num_output_placeholders += (
self.num_sampled_tokens_per_step + cur_num_spec_tokens
)What the data looks like now (illustrative ids):
SchedulerOutput(
scheduled_new_reqs=[],
scheduled_cached_reqs=CachedRequestData(
req_ids=["B", "A"],
new_block_ids=[None, ([b512, ..., b1023],)], # 512 new block ids for A
num_computed_tokens=[540, 8191], # values before this step
num_output_tokens=[41, 0], ...), # 40 known + 1 placeholder for B
num_scheduled_tokens={"B": 1, "A": 8191},
total_num_scheduled_tokens=8192,
scheduled_spec_decode_tokens={}, finished_req_ids=set(), ...)
B.num_computed_tokens = 541; B.num_output_placeholders = 2 # tokens 41 and 42 in flight
A.num_computed_tokens = 16382; A.is_prefill_chunk = Truenum_output_tokens includes the placeholders (vllm/v1/core/sched/scheduler.py:1825-1827). The page Data contracts between the parts of vLLM lists all fields.
4The executor starts execute_model, without a wait#
Back in step_with_batch_queue, the engine core passes the plan to the executor:
with self.log_error_detail(scheduler_output):
exec_future = self.model_executor.execute_model(
scheduler_output, non_block=True
)
if not self.is_mm_encoder_only:
model_executed = scheduler_output.total_num_scheduled_tokens > 0model_executed is true, because 8192 tokens are scheduled. The uni executor runs the worker method at once, in this thread:
def execute_model( # type: ignore[override]
self, scheduler_output: SchedulerOutput, non_block: bool = False
) -> ModelRunnerOutput | None | Future[ModelRunnerOutput | None]:
output = self.collective_rpc(
"execute_model",
args=(scheduler_output,),
non_block=non_block,
single_value=True,
)
# In non-blocking mode, surface any exception as early as possible.
if non_block and output.done():
# Raise the exception in-line if the task failed.
output.result()
return outputThe call goes through WorkerWrapperBase.execute_model (vllm/v1/worker/worker_base.py:378-383) to Worker.execute_model, which calls the model runner:
with self.annotate_profile(scheduler_output):
output = self.model_runner.execute_model(
scheduler_output, intermediate_tensors
)With one GPU there is no message queue. With TP = 2, this stop would put the call into a shared-memory queue for two worker processes instead (Executors and workers: from the engine core to the GPUs).
5Model Runner V2: update states, sort the batch, pick a CUDA graph mode#
We are in vllm/v1/worker/gpu/model_runner.py. Model Runner V2 keeps its own copy of each request's state. First it applies the changes from the plan:
if not dummy_run:
# Update the request states.
self.update_pp_decode_requests()
self.finish_requests(scheduler_output)
self.free_states(scheduler_output)
self.add_requests(scheduler_output)
self.update_requests(scheduler_output)
self.block_tables.apply_staged_writes()update_requests stores the computed-token counts and appends A's 512 new block ids to its block table (vllm/v1/worker/gpu/model_runner.py:1257-1269). Then the runner sorts the batch: decodes first, then prefills (vllm/v1/worker/gpu/model_runner.py:1312-1314, vllm/v1/worker/gpu/model_runner.py:2471-2483). B has 1 token, which equals the decode query length, so B is row 0 and A is row 1.
Next it picks a CUDA graph mode for the batch shape (vllm/v1/worker/gpu/model_runner.py:1806-1821). Our batch has 8192 tokens and contains a prefill. On an H100 the largest captured size is 512 tokens: min(max_num_seqs × 1 × 2, 512) with max_num_seqs = 1024 (vllm/config/vllm.py:2439-2449). No captured graph matches 8192 tokens, so dispatch returns mode NONE: eager (vllm/v1/worker/gpu/cudagraph_utils.py:535-553). With DP = 1, no padding across ranks changes this result (vllm/v1/worker/gpu/dp_utils.py:270-279).
6prepare_inputs and prepare_attn: the flat batch#
The runner packs both requests into one flat batch of 8192 tokens, as in Input metadata. query_start_loc marks where each request starts:
# Get query_start_loc.
# num_reqs_padded is None for PIECEWISE graphs (no request padding needed)
num_reqs_padded = batch_desc.num_reqs or num_reqs
query_start_loc_np = np.empty(self.max_num_reqs + 1, dtype=np.int32)
query_start_loc_np[0] = 0
np.cumsum(num_scheduled_tokens_np, out=query_start_loc_np[1 : num_reqs + 1])
# Pad for full CUDA graph mode.
# Some attention backends like FA3 require query_start_loc to be non-decreasing.
query_start_loc_np[num_reqs + 1 :] = num_tokens
query_start_loc = self.input_buffers.query_start_loc
async_tensor_h2d(query_start_loc_np, out=query_start_loc)A is in a prefill, so its prompt tokens are copied on the GPU from the request state (vllm/v1/worker/gpu/model_runner.py:1469-1479). B's input token is token 41, which the CPU does not know yet. It comes from last_sampled_tokens on the GPU, which step k-1 will write before step k runs on the same stream:
# Some input token ids are directly read from the last sampled tokens
# and draft tokens. Also, get the logits indices to sample tokens from.
logits_indices = combine_sampled_and_draft_tokens(
self.input_buffers.input_ids,
idx_mapping,
self.req_states.last_sampled_tokens,
query_start_loc,
seq_lens,
self.req_states.prefill_len.gpu,
self.req_states.draft_tokens,
cu_num_logits,
total_num_logits,
self.model_state.num_new_sampled_tokens_per_step,
)Without speculative decoding, each request gets one logits row (vllm/v1/worker/gpu/model_runner.py:1401-1408): the last token of its query. Then prepare_attn builds the block tables and the slot of each new token, and the model state builds the attention metadata (vllm/v1/worker/gpu/model_runner.py:1838, vllm/v1/worker/gpu/model_runner.py:1940-1951).
What the data looks like now (step k, derived from our numbers):
num_scheduled_tokens = [1, 8191] # row 0 = B, row 1 = A
query_start_loc int32 [3] = [0, 1, 8192]
positions int64 [8192] = [540, 8191, 8192, ..., 16381]
seq_lens int32 [2] = [541, 16382]
input_ids int32 [8192] = [token 41 of B (from GPU), A's prompt tokens 8191..16381]
logits_indices [2] = [0, 8191]
slot_mappings one slot per token: B -> its block index 33, offset 12 (540 = 33 x 16 + 12)
A -> its block indexes 511 .. 1023 (positions 8191 .. 16381)The block indexes are positions in each request's own block table. The table maps them to physical block ids in the block pool, and the slot of a token is block_id × 16 + offset (vllm/v1/worker/gpu/block_table.py:409).
7The forward pass is queued, and execute_model returns None#
In mode NONE, the runner calls the model eagerly, inside the forward context that gives each attention layer its metadata:
with set_forward_context(
attn_metadata,
self.vllm_config,
num_tokens=input_batch.num_tokens_after_padding,
cudagraph_runtime_mode=batch_desc.cg_mode,
num_tokens_across_dp=dp_sync_state
and dp_sync_state.num_tokens_across_dp,
batch_descriptor=batch_descriptor,
ubatch_slices=ubatch_slices,
slot_mapping=slot_mappings_by_layer,
skip_compiled=skip_compiled,
is_padding=input_batch.is_padding,
):
self.kv_connector.pre_forward(scheduler_output, input_batch)
if ubatch_state is not None:
assert self.ubatch_runner is not None
model_output = self.ubatch_runner.run(
self.model, model_inputs, ubatch_state
)
elif batch_desc.cg_mode == CUDAGraphMode.PIECEWISE:
# Run the PIECEWISE graph (compiled PW cudagraph or breakable
# cudagraph, chosen inside run_pw_graph). cg_mode is only
# PIECEWISE after the cudagraph manager exists.
assert self.cudagraph_manager is not None
model_output = self.cudagraph_manager.run_pw_graph(
self.model, model_inputs
)
else:
# Eager (NONE): call the raw model directly.
model_output = self.model(**model_inputs)The Python code returns as soon as all kernels are queued; the GPU runs them later, after step k-1. The runner keeps the hidden states for the sampler and returns None:
finished_req_ids = scheduler_output.finished_req_ids
self.execute_model_state = ExecuteModelState(
input_batch=input_batch,
attn_metadata=attn_metadata,
slot_mappings_by_layer=slot_mappings_by_layer,
hidden_states=hidden_states,
aux_hidden_states=aux_hidden_states,
dp_sync_state=dp_sync_state,
finished_req_ids=finished_req_ids,
ec_connector_output=ec_connector_output,
cudagraph_stats=cudagraph_stats,
num_spec_tokens_to_schedule=scheduler_output.num_spec_tokens_to_schedule,
)
if not self.is_last_pp_rank:
# Non-last PP rank: return IntermediateTensors for sending.
assert output_intermediate_tensors is not None
assert self.pp_handler is not None
return self.pp_handler.relay_aux_hidden_states(
model_inputs["intermediate_tensors"], output_intermediate_tensors
)
return NoneBack in the uni executor, the future is already done with the value None (vllm/v1/executor/uniproc_executor.py:112-113). The tour Tour: One forward pass goes inside the model call.
8The grammar bitmask: not needed this time#
Back in step_with_batch_queue. Before sampling, the engine core may compute a grammar bitmask for structured output requests, on the CPU, while the GPU works:
if self.is_pooling_model or not model_executed:
# No sampling required (no requests scheduled).
future = cast(Future[ModelRunnerOutput], exec_future)
else:
if not scheduler_output.pending_structured_output_tokens:
# We aren't waiting for any tokens, get any grammar output
# and sample immediately.
grammar_output = self.scheduler.get_grammar_bitmask(
scheduler_output
)
future = self.model_executor.sample_tokens(
grammar_output, non_block=True
)
else:
# We need to defer sampling until we have processed the model output
# from the prior step.
deferred_scheduler_output = scheduler_outputOur requests do not use structured output, so has_structured_output_requests is false and get_grammar_bitmask returns None at once (vllm/v1/core/sched/scheduler.py:2091-2092). If B used a JSON schema, its token 41 would still be in flight. Then pending_structured_output_tokens would be true, and sampling would wait until stop 11 (vllm/v1/core/sched/async_scheduler.py:37-39, vllm/v1/engine/core.py:792-810).
9sample_tokens: logits for two rows, one token, an AsyncOutput#
sample_tokens(None, non_block=True) takes the same path as stop 4 and reaches the runner. It computes logits only for the two rows in logits_indices, and samples:
sample_hidden_states = hidden_states[input_batch.logits_indices]
logits = self.model.compute_logits(sample_hidden_states) # 1 sampled token per request, except chunked-prefill requests
# (seq_len < prefill_len) which aren't done prefilling and produce no
# output token. num_rejected is always 0 here (one logit per request).
num_sampled, num_rejected = get_num_sampled_and_rejected(
input_batch.seq_lens.new_ones(input_batch.num_reqs),
input_batch.seq_lens,
input_batch.cu_num_logits,
input_batch.idx_mapping,
self.req_states.prefill_len.gpu,
)The comment gives the rule: a request in a chunked prefill "produce[s] no output token". So num_sampled = [1, 0]: B gets token 42, A gets nothing. The runner starts the copy of the result to the CPU on a separate stream. Then it writes token 42 into last_sampled_tokens for step k+1:
# Start async output copy here so that it can overlap with speculator proposal.
async_output = AsyncOutput(
model_runner_output=model_runner_output,
sampler_output=sampler_output,
num_sampled_tokens=num_sampled,
main_stream=self.main_stream,
copy_stream=self.output_copy_stream,
check_ep_fault=self.check_ep_fault,
pending_aux_output=pending_aux_output,
) # Postprocess results and update request states.
# NOTE: This is intentionally done after creating the AsyncOutput,
# ensuring that `copy_event` is recorded before calling postprocess.
# This sequencing may slightly reduce latency as async D2H copy does not
# need to wait for the postprocess to finish.
self.postprocess_sampled(
input_batch.idx_mapping,
sampler_output.sampled_token_ids,
num_sampled,
num_rejected,
input_batch.query_start_loc,
)The uni executor wraps the AsyncOutput in an AsyncOutputFuture; nothing waits yet (vllm/v1/executor/uniproc_executor.py:109-111). The sampling methods are in Sampling.
What the data looks like now (on the GPU, step k):
logits [2, vocab_size] # rows of B and A
sampled_token_ids [2, 1] = [[t42], [x]] # x is ignored
num_sampled [2] = [1, 0]
last_sampled_tokens[B] = t42 # input of B in step k+110The batch queue is full: pop step k-1 and wait#
Step k goes into the queue. Now the queue has 2 entries, which is full, so the function does not return early:
if not deferred_scheduler_output:
# Add this step's future to the queue.
batch_queue.appendleft((future, scheduler_output, exec_future))
if len(batch_queue) < self.batch_queue_size and (
model_executed or self.scheduler.has_requests()
):
# Don't block on next worker response unless the queue is full
# or there are no more requests to schedule.
return None, model_executed # Block until the next result is available.
future, scheduler_output, exec_model_fut = batch_queue.pop()
with (
self.capture_iteration_details(scheduler_output) as iteration_details,
self.log_error_detail(scheduler_output),
):
model_output = future.result()
if model_output is None:
# None from sample_tokens() implies that the original execute_model()
# call failed - raise that exception.
exec_model_fut.result()
raise RuntimeError("unexpected error")pop() takes step k-1 from the right end. future.result() calls AsyncOutput.get_output() (vllm/v1/executor/uniproc_executor.py:38-48). This is the one place where the engine core waits for the GPU:
def get_output(self) -> ModelRunnerOutput:
self.copy_event.synchronize()
# NOTE(woosuk): The following code is to ensure compatibility with
# the existing model runner.
# Going forward, we should keep the data structures as NumPy arrays
# rather than Python lists.
sampled_token_ids: list[list[int]] = self.sampled_token_ids.tolist()
num_sampled_tokens: list[int] = self.num_sampled_tokens_np.tolist()
for token_ids, num_tokens in zip(sampled_token_ids, num_sampled_tokens):
del token_ids[num_tokens:]
self.model_runner_output.sampled_token_ids = sampled_token_idsIf the GPU has already finished step k-1, the wait returns at once. Meanwhile, step k is already queued behind it on the GPU. Line 183 trims each row to its sampled count, so A's row becomes an empty list.
What the data looks like now (step k-1):
ModelRunnerOutput(req_ids=["B", "A"],
sampled_token_ids=[[t41], []], # A ran its first chunk in step k-1
logprobs=None, ...)11update_from_output for step k-1#
Before the update, the engine core applies any aborts that arrived during the step (vllm/v1/engine/core.py:780-785). Then the scheduler walks over the requests of step k-1:
# Check for stop and update request status.
if new_token_ids:
new_token_ids, stopped = self._update_request_with_output(
request, new_token_ids, is_stale=output_is_stale
)- B. It has a new token, so
_update_request_with_outputappends t41 and runscheck_stop(vllm/v1/core/sched/scheduler.py:2598-2603). t41 is not EOS, so B keeps running.AsyncSchedulerremoves one placeholder: 2 becomes 1 (vllm/v1/core/sched/async_scheduler.py:71-73). It caches full blocks up to541 - 1 = 540tokens for the prefix cache (lines 76-79 there). - A. No new tokens, no pooler output, no stop: the scheduler makes no output for A (
vllm/v1/core/sched/scheduler.py:2293-2295).
The outputs are grouped by front end (vllm/v1/core/sched/scheduler.py:2486-2491). With stats on, the scheduler stats go into the same object (vllm/v1/core/sched/scheduler.py:2507-2521).
What the data looks like now:
{0: EngineCoreOutputs(
outputs=[EngineCoreOutput(request_id="B", new_token_ids=[t41], finish_reason=None, ...)],
scheduler_stats=SchedulerStats(num_running_reqs=2, num_waiting_reqs=0, ...))}
B: num_tokens = 541, num_computed_tokens = 541, num_output_placeholders = 1 # t42 still in flight12The outputs go to the output thread#
step_with_batch_queue returns the dict and model_executed = True. _process_engine_step puts each pair into output_queue (stop 1, lines 1673-1674) and calls post_step, which does nothing with async scheduling (vllm/v1/engine/core.py:689-696). The output thread takes the pair, encodes it with msgpack, and sends it on the PUSH socket of front end 0:
buffer = reuse_buffers.pop() if reuse_buffers else bytearray()
buffers = encoder.encode_into(outputs, buffer)
tracker = self._send_msg_tracking_payload(
sockets[client_index], buffers
)From here, Tour 1 continues: the front end detokenizes t41 and writes one SSE chunk (Tour 1, stop 15). The busy loop goes back to stop 1. The next call schedules step k+1: B again 1 token, and A its last chunk of 20000 - 16382 = 3618 tokens. That chunk ends the prompt, so A samples its first token there.
Deep dive#
Deep diveThe counters of B and A across three calls
Values right after each event. "S k" = schedule step k; "U k" = update with the result of step k. (Derived from the rules in stops 2, 3 and 11.)
| Event | B: num_tokens / computed / placeholders | A: computed / prefill chunk? | Scheduled (B, A) |
|---|---|---|---|
| before S k | 540 / 540 / 1 | 8191 / yes | - |
| S k (this tour) | 540 / 541 / 2 | 16382 / yes | (1, 8191) |
| U k-1 (this tour) | 541 / 541 / 1 | 16382 / yes | - |
| S k+1 | 541 / 542 / 2 | 20000 / no; A gets 1 placeholder | (1, 3618) |
| U k | 542 / 542 / 1 | 20000 / no | - |
| S k+2 | 542 / 543 / 2 | 20001 / no; 2 placeholders | (1, 1) |
| U k+1 | 543 / 543 / 1 | A gets its first token | - |
In S k+1, A's need is 20000 + 0 - 16382 = 3618, so the budget is not full: 8192 − 1 − 3618 = 4573 tokens remain for waiting requests. In S k+2, A decodes for the first time, with its first token still in flight.
Deep diveWhere the time goes in this call
- Stops 2-3 run in Python in the scheduler. Their cost grows with the number of running requests, not with the number of tokens. The comment in
update_from_outputwarns that its loop "can be a performance bottleneck" with 1K or more requests (vllm/v1/core/sched/scheduler.py:2163-2165). - Stops 5-7 and 9 run on the CPU but only queue GPU work. Their cost is the Python code of the model runner and the kernel launches. A CUDA graph replaces many launches with one; our eager step does not have this help.
- Stop 10 is the only wait for the GPU. When the GPU step takes longer than all the CPU work of a call, the main thread waits here, and the GPU never waits. That is the goal of async scheduling.
- Stop 12 encoding and sending run in the output thread, in parallel with the next call.
Deep diveWhat changes with TP = 2
Stops 1-3, 8 and 10-12 do not change. In stops 4 and 9, the multiproc executor writes the call once into the shared-memory queue. Both worker processes run stops 5-7 and 9 on their own GPU, and they meet in NCCL all-reduces. Only worker 0 replies (vllm/v1/executor/multiproc_executor.py:549-563). The wait of stop 10 then happens in worker 0's WorkerAsyncOutputCopy thread, which calls get_output(), and the engine core waits for the reply message (vllm/v1/executor/multiproc_executor.py:1000-1002, vllm/v1/executor/multiproc_executor.py:1019-1039). See Executors and workers: from the engine core to the GPUs and Tour: One step across GPUs.
What can go wrong#
- A long prompt slows every stream. In our step, B shares a step of 8192 tokens with A's chunk. B's next token comes only when that large step ends.
long_prefill_token_thresholdlimits one prompt's share of a step (vllm/v1/core/sched/scheduler.py:764-765). - No KV blocks for the chunk. If
allocate_slotsfails, the scheduler preempts a running request to free blocks. With the default first-come-first-served policy, the victim is the last request inself.running(vllm/v1/core/sched/scheduler.py:850-858,vllm/v1/core/sched/scheduler.py:889-900). In our batch this is A itself: A loses its KV cache and must compute it again later, and B keeps running. - An exception in the model.
log_error_detaildumps theSchedulerOutputof the failed step, and the engine core dies withENGINE_CORE_DEAD(vllm/v1/engine/core.py:561-575, The engine core loop: threads, queues and the busy loop). - A request aborted during the step. Its row still runs on the GPU.
_process_aborts_queuefinishes it before the update, and the update skips it (vllm/v1/engine/core.py:780-782,vllm/v1/core/sched/scheduler.py:2184-2192).
Why it matters. Every performance number of vLLM comes from this loop. The size of each step comes from stop 2, the CPU cost from stops 2-9, and the GPU cost from stop 7. When you tune max_num_batched_tokens or read a profile, you can now map each part to a stop.
Check yourself#
- 1In this tour, which step's token does the call send to the front end?easy
Show answer
The token of step k-1 (B's token 41). The call schedules and starts step k first, then pops step k-1 from the batch queue and updates the scheduler with it (
vllm/v1/engine/core.py:753-785). - 2Why does A get 8191 tokens and not 11809?medium
Show answer
The running pass caps each request at the remaining token budget (
vllm/v1/core/sched/scheduler.py:766-768). B came first and took 1 token of the 8192, so 8191 remained.long_prefill_token_thresholdis 0 by default, so it does not cap A further (vllm/config/scheduler.py:79-84). - 3Where does B's input token for step k come from, given that the CPU has not received it yet?medium
Show answer
From
req_states.last_sampled_tokenson the GPU.combine_sampled_and_draft_tokenscopies it intoinput_idsinprepare_inputs(vllm/v1/worker/gpu/model_runner.py:1493-1500). Step k-1 writes it inpostprocess_sampled, earlier on the same CUDA stream (vllm/v1/worker/gpu/model_runner.py:1715-1726). - 4A has a logits row in step k. Why does it have no token in thehard
ModelRunnerOutput, and why does the scheduler send nothing for A?Show answer
Without spec decoding, every request gets one logits row (
vllm/v1/worker/gpu/model_runner.py:1401-1408). The sampler setsnum_sampled = 0for a chunked-prefill request (vllm/v1/worker/gpu/sample/sampler.py:190-199), andget_output()trims the row to an empty list (vllm/v1/worker/gpu/async_utils.py:182-183). With no new tokens, no pooler output and no stop, the scheduler makes noEngineCoreOutput(vllm/v1/core/sched/scheduler.py:2293-2295). - 5Which CUDA graph mode does step k use, and what would a step with only B and A decoding use?hard
Show answer
Step k has 8192 tokens and a prefill. The largest captured size on this GPU is 512 (
vllm/config/vllm.py:2439-2449), so no captured graph matches, anddispatchreturnsNONE(eager) (vllm/v1/worker/gpu/cudagraph_utils.py:535-553). A step with two 1-token decodes is a uniform decode batch of 2 tokens. So it can replay a FULL graph under the default mode FULL_AND_PIECEWISE (vllm/config/vllm.py:310).