Async scheduling: overlap the CPU and the GPU

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

Each engine step has CPU work and GPU work. The CPU schedules the batch, prepares the inputs, launches the kernels, and later turns the sampled tokens into outputs. The GPU runs the forward pass and the sampler. If the CPU waits for each step before it plans the next one, the GPU is idle during the CPU work. For small decode batches, that CPU work can take a large part of the step time.

The fix is to plan one step ahead. DeepInfer uses the same idea: the CPU prepares batch N+1 while the GPU runs batch N (Zero-overhead scheduling↗). vLLM does it with three parts:

1. The engine core keeps a batch queue of started steps, and it waits only for the oldest one (The engine core loop: threads, queues and the busy loop).
2. AsyncScheduler counts an output placeholder for each token that a started step will sample. So it can schedule the next decode step before it knows the token.
3. Model Runner V2 keeps the last sampled token of each request in a GPU tensor. The next step reads its input token from there, without a copy to the CPU and back.

An everyday comparison. Think of a relay race where the next runner starts to run before the baton arrives, and takes it at full speed. The next step starts before the token arrives. The comparison stops in one place: in a relay race the baton must arrive. Here the token never has to reach the CPU before the next step: the GPU passes it to itself.

The picture#

Compare the GPU lanes of panels 1 and 2. Without async scheduling, the GPU has a gap in every step. With it, the GPU lane is full, and the CPU waits instead (dashed blocks). Panel 3 shows why the GPU can start step N+1 at once: it takes its input token from its own memory.

In the code#

Step 1: when async scheduling is on#

The flag async_scheduling has the default None (vllm/config/scheduler.py:211-214). VllmConfig.__post_init__ turns None into True unless an incompatible option is set:

        elif self.scheduler_config.async_scheduling is None:
            # Enable async scheduling unless there is an incompatible option.
            if (
                self.model_config is not None
                and self.model_config.runner_type == "pooling"
            ):
                # The current implementation of asynchronous scheduling negatively
                # impacts performance of pooling models, so we disable by default.
                logger.debug(
                    "Disabling asynchronous scheduling by default for pooling model."
                )
                self.scheduler_config.async_scheduling = False
            elif not executor_supports_async_sched:
                logger.warning_once(
                    "Async scheduling will be disabled because it is not supported "
                    "with the `%s` distributed executor backend. ",
                    executor_backend,
                )
                self.scheduler_config.async_scheduling = False
            elif uses_rocm_deepep_ht_dbo:
                logger.warning_once(
                    "Async scheduling is disabled for ROCm DeepEP "
                    "high-throughput DBO because that combination can corrupt "
                    "DP+EP generation accuracy."
                )
                self.scheduler_config.async_scheduling = False
            elif v1_async_sched_incompat is not None:
                logger.warning_once(
                    "Async scheduling is disabled because the V1 model runner "
                    "does not support it with %s.",
                    v1_async_sched_incompat,
                )
                self.scheduler_config.async_scheduling = False
            else:
                self.scheduler_config.async_scheduling = True

So async scheduling is off in these cases:

• You pass --no-async-scheduling (the flag is --async-scheduling, vllm/engine/arg_utils.py:1800-1802).
• The model is a pooling model, for example an embedding model (vllm/config/vllm.py:1666-1675).
• The executor class does not support it (vllm/config/vllm.py:1676-1682). The base class says no (vllm/v1/executor/abstract.py:452-455). The uni and mp executors say yes (vllm/v1/executor/uniproc_executor.py:156-158, vllm/v1/executor/multiproc_executor.py:565-567). The Ray executor inherits the mp answer (vllm/v1/executor/ray_executor_v2.py:239-247). An out-of-tree executor class gets False unless it overrides the method.
• ROCm with dual-batch overlap and the deepep_high_throughput all-to-all backend (vllm/config/vllm.py:1624-1628, vllm/config/vllm.py:1683-1689).
• The fallback Model Runner V1 is in use, with one of three options: pipeline parallelism, disable_padded_drafter_batch, or an unsupported speculative method (vllm/config/vllm.py:1630-1645, vllm/config/vllm.py:1690-1696). The supported methods are the EAGLE types, GPU n-gram, draft_model, dspark and dflash. With the default Model Runner V2, none of these options turn it off.
• The CPU platform. Its platform hook always sets async_scheduling = False (vllm/platforms/cpu.py:342-343). The hook runs later in __post_init__, so it wins over your flag and raises no error (vllm/config/vllm.py:2046-2048).

If you set --async-scheduling yourself and the executor, ROCm or Model Runner V1 case applies, the config raises ValueError instead (vllm/config/vllm.py:1647-1663). A pooling model does not raise: async scheduling stays on. With async scheduling on and no scheduler class set, the scheduler class is AsyncScheduler (vllm/config/scheduler.py:231-236). The batch queue has pp_size + 1 slots with Model Runner V2:

    @property
    def max_concurrent_batches(self) -> int:
        # PP requires PP-size concurrent batches to fill the pipeline.
        # Async scheduling requires 2 concurrent batches to overlap.
        pp_size = self.parallel_config.pipeline_parallel_size
        if self.scheduler_config.async_scheduling:
            if self.use_v2_model_runner:
                return pp_size + 1
            # V1 Model Runner does not fully support async scheduling with PP.
            if pp_size <= 1:
                return 2
        return pp_size

Step 2: AsyncScheduler and the placeholders#

AsyncScheduler is a small subclass of Scheduler. It changes two hooks. After each schedule(), it adds placeholders:

    def _update_after_schedule(self, scheduler_output: SchedulerOutput) -> None:
        super()._update_after_schedule(scheduler_output)
        spec_decode_tokens = scheduler_output.scheduled_spec_decode_tokens
        # Use the latest num of scheduled draft tokens in next step as placeholder.
        self._spec_token_placeholders = [
            -1
        ] * scheduler_output.num_spec_tokens_to_schedule
        for req_id in scheduler_output.num_scheduled_tokens:
            request = self.requests[req_id]
            if request.is_prefill_chunk:
                continue

            scheduler_output.pending_structured_output_tokens |= (
                request.use_structured_output and request.num_output_placeholders > 0
            )
            # The request will generate num_sampled_tokens_per_step new tokens
            # plus num_spec_tokens in this scheduling step. Diffusion has no AR
            # bonus token (num_sampled_tokens_per_step == 0) — only the canvas
            # (spec) tokens.
            cur_num_spec_tokens = len(spec_decode_tokens.get(req_id, ()))
            request.num_output_placeholders += (
                self.num_sampled_tokens_per_step + cur_num_spec_tokens
            )
            # Add placeholders for the new draft/spec tokens.
            # We will update the actual spec token ids in the worker process.
            request.spec_token_ids = self._spec_token_placeholders

1. Lines 34-35: a request in the middle of a chunked prefill samples no token, so it gets no placeholder.
2. Lines 37-39: a structured output request with a token still in flight sets pending_structured_output_tokens (Step 5).
3. Lines 45-47: each sampling step adds one placeholder per sampled token, plus one per draft token with speculative decoding.
4. Line 50: the draft tokens of the next step are not known yet either. The scheduler uses a list of -1 values as their placeholders (vllm/v1/core/sched/async_scheduler.py:17-18, vllm/v1/core/sched/async_scheduler.py:28-31). The worker fills in the real draft tokens on the GPU.

The running pass of schedule() counts the placeholders as tokens that the request already has:

            num_new_tokens = (
                request.num_tokens_with_spec
                + request.num_output_placeholders
                - request.num_computed_tokens
            )

Take a decode with one token in flight and no speculative decoding. The request has T known tokens, one placeholder, and num_computed_tokens = T. The result is T + 1 - T = 1: the scheduler plans exactly one more token. When the real token arrives, the second hook removes the placeholder:

    def _update_request_with_output(
        self, request: Request, new_token_ids: list[int], is_stale: bool = False
    ) -> tuple[list[int], bool]:
        status_before_update = request.status
        # Tokens trimmed at a stop resolve their placeholders too.
        num_new_tokens = len(new_token_ids)
        new_token_ids, stopped = super()._update_request_with_output(
            request, new_token_ids, is_stale
        )

        # Placeholders were zeroed at preemption; a stale delivery must not
        # decrement them (it would underflow).
        if not is_stale:
            request.num_output_placeholders -= num_new_tokens
            assert request.num_output_placeholders >= 0

        # Cache the new tokens. Preempted requests should be skipped.
        if status_before_update == RequestStatus.RUNNING:
            self.kv_cache_manager.cache_blocks(
                request, request.num_computed_tokens - request.num_output_placeholders
            )
        return new_token_ids, stopped

The tour Tour 1, the loop shows the counters over three steps.

Step 3: the batch queue gives the overlap#

The engine core uses step_with_batch_queue when max_concurrent_batches > 1 (vllm/v1/engine/core.py:205-211, vllm/v1/engine/core.py:227-229). On each call it schedules and starts step N+1, and only then waits for step N:

                batch_queue.appendleft((future, scheduler_output, exec_future))
                if len(batch_queue) < self.batch_queue_size and (
                    model_executed or self.scheduler.has_requests()
                ):
                    # Don't block on next worker response unless the queue is full
                    # or there are no more requests to schedule.
                    return None, model_executed

        # Block until the next result is available.
        future, scheduler_output, exec_model_fut = batch_queue.pop()
        with (
            self.capture_iteration_details(scheduler_output) as iteration_details,
            self.log_error_detail(scheduler_output),
        ):
            model_output = future.result()

Timeline of one call, for steps N and N+1:

1. The GPU runs step N (it was queued in the previous call).
2. The CPU schedules step N+1. The decode requests of step N+1 count their step-N token as a placeholder.
3. The CPU runs the CPU part of the model runner for step N+1. The kernels of step N+1 go into the same CUDA stream, behind step N.
4. The CPU waits for the result of step N. Often the GPU is already done, so the wait is short.
5. The CPU runs update_from_output for step N and sends its tokens. The GPU already runs step N+1.

The page The engine core loop: threads, queues and the busy loop reads the whole function line by line.

Step 4: the next input token comes from the GPU#

The CPU does not know the token of step N when it prepares step N+1. Model Runner V2 does not need it. The sampler output stays on the GPU, and postprocess_sampled writes it into req_states.last_sampled_tokens with the post_update kernel:

    def postprocess_sampled(
        self,
        idx_mapping: torch.Tensor,  # May include -1 for masked entries
        sampled_tokens: torch.Tensor,
        num_sampled: torch.Tensor,
        num_rejected: torch.Tensor,
        query_start_loc: torch.Tensor | None = None,
    ) -> None:
        # Update the number of computed tokens.
        output_bin_counts = None
        if self.is_last_pp_rank:
            assert self.sampler is not None
            output_bin_counts = self.sampler.penalties_state.output_bin_counts
        post_update(
            idx_mapping,
            self.req_states.num_computed_tokens.gpu,
            self.req_states.last_sampled_tokens,
            output_bin_counts,
            sampled_tokens,
            num_sampled,
            num_rejected,
            query_start_loc,
            self.req_states.all_token_ids.gpu,
            self.req_states.total_len.gpu,
        )

In the next step, prepare_inputs copies that token into the input ids on the GPU:

        # Some input token ids are directly read from the last sampled tokens
        # and draft tokens. Also, get the logits indices to sample tokens from.
        logits_indices = combine_sampled_and_draft_tokens(
            self.input_buffers.input_ids,
            idx_mapping,
            self.req_states.last_sampled_tokens,
            query_start_loc,
            seq_lens,
            self.req_states.prefill_len.gpu,
            self.req_states.draft_tokens,
            cu_num_logits,
            total_num_logits,
            self.model_state.num_new_sampled_tokens_per_step,
        )

The copy to the CPU runs at the same time on a separate CUDA stream. AsyncOutput starts it in sample_tokens and records an event (vllm/v1/worker/gpu/async_utils.py:171); nobody waits for it there:

        with stream(copy_stream, main_stream):
            copy_stream.wait_stream(main_stream)

            self.sampled_token_ids = async_copy_to_np(sampler_output.sampled_token_ids)

    def get_output(self) -> ModelRunnerOutput:
        self.copy_event.synchronize()

        # NOTE(woosuk): The following code is to ensure compatibility with
        # the existing model runner.
        # Going forward, we should keep the data structures as NumPy arrays
        # rather than Python lists.
        sampled_token_ids: list[list[int]] = self.sampled_token_ids.tolist()
        num_sampled_tokens: list[int] = self.num_sampled_tokens_np.tolist()
        for token_ids, num_tokens in zip(sampled_token_ids, num_sampled_tokens):
            del token_ids[num_tokens:]
        self.model_runner_output.sampled_token_ids = sampled_token_ids

The only wait is copy_event.synchronize() in get_output() (line 174). With the uni executor, the engine core calls it when it pops step N from the batch queue (vllm/v1/executor/uniproc_executor.py:38-48). With the mp executor, a worker thread calls it (Step 6). The design doc of Model Runner V2 states the rule: the core execution loop is "a CUDA stream with no CPU synchronization points" (docs/design/model_runner_v2.md:47). get_output also trims each row to the number of sampled tokens (line 183). So a prefill chunk that sampled nothing gets an empty list.

Step 5: execute_model and sample_tokens are two calls#

The engine core calls the workers twice per step: execute_model runs the forward pass, and sample_tokens computes logits and samples. The worker interface links this design to structured outputs:

    def execute_model(
        self, scheduler_output: SchedulerOutput
    ) -> ModelRunnerOutput | AsyncModelRunnerOutput | None:
        """If this method returns None, sample_tokens should be called immediately after
        to obtain the ModelRunnerOutput.

        Note that this design may be changed in future if/when structured outputs
        parallelism is re-architected.
        """
        raise NotImplementedError

    def sample_tokens(
        self, grammar_output: GrammarOutput
    ) -> ModelRunnerOutput | AsyncModelRunnerOutput:
        """Should be called immediately after execute_model iff it returned None."""
        raise NotImplementedError

Between the two calls the engine core computes the grammar bitmask for structured output on the CPU, while the GPU runs the forward pass:

                if not scheduler_output.pending_structured_output_tokens:
                    # We aren't waiting for any tokens, get any grammar output
                    # and sample immediately.
                    grammar_output = self.scheduler.get_grammar_bitmask(
                        scheduler_output
                    )
                    future = self.model_executor.sample_tokens(
                        grammar_output, non_block=True
                    )
                else:
                    # We need to defer sampling until we have processed the model output
                    # from the prior step.
                    deferred_scheduler_output = scheduler_output

The bitmask of step N+1 needs the token of step N, because the grammar must first advance by that token. When a structured output request still has a token in flight, the token is not on the CPU yet. Then the engine defers sampling: it processes step N first, computes the bitmask, and only then queues sample_tokens for step N+1 (vllm/v1/engine/core.py:789-810). In this case the GPU can wait for the CPU for a short time. Structured outputs as token masks are explained in Structured outputs↗.

Step 6: the same overlap with worker processes#

With the uni executor, execute_model and sample_tokens run in the engine core's main thread. They return without a GPU wait; non_block only moves the wait into a future (vllm/v1/executor/uniproc_executor.py:108-117). With the mp executor, each worker process has an extra thread, WorkerAsyncOutputCopy, only when async scheduling is on (vllm/v1/executor/multiproc_executor.py:690-699). On the output rank, the worker's main thread hands each output to it and reads the next call at once (vllm/v1/executor/multiproc_executor.py:1019-1020, vllm/v1/executor/multiproc_executor.py:1070-1071). The extra thread calls get_output() and writes the result into the reply queue (vllm/v1/executor/multiproc_executor.py:1000-1002, vllm/v1/executor/multiproc_executor.py:1037-1039). See Executors and workers: from the engine core to the GPUs.

Step 7: what it costs#

• One step of extra work after a stop. When step N samples EOS or a stop token id, step N+1 for that request is already on the GPU. update_from_output skips results of finished requests (vllm/v1/core/sched/scheduler.py:2184-2192). The GPU time of that row is lost, but the client never sees the token.
• No extra step for max_tokens. The running pass skips a request when the step in flight surely reaches max_tokens (vllm/v1/core/sched/scheduler.py:720-734).
• Stop strings cost more. The engine core has no text. The front end finds a stop string after detokenization and sends an ABORT (vllm/v1/engine/output_processor.py:825-830, vllm/v1/engine/output_processor.py:878-882, vllm/v1/engine/async_llm.py:918-921). By then the engine may have started more than one further step for the request.
• Prefix caching lags by one step. After an update, the scheduler caches only blocks up to num_computed_tokens - num_output_placeholders, the tokens whose values are known (vllm/v1/core/sched/async_scheduler.py:75-79).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

What can go wrong#

• Async scheduling is off and you do not know it. For a pooling model it is off by default and the log says so only at DEBUG level (vllm/config/vllm.py:1672-1675). For an unsupported executor it logs a warning (vllm/config/vllm.py:1677-1681).
• A custom executor loses the overlap. An out-of-tree executor class gets supports_async_scheduling() = False from the base class (vllm/v1/executor/abstract.py:452-455). Override it if your executor returns futures correctly.
• Structured output slows decode. Each step with a pending structured output token defers sampling until the previous step is processed (vllm/v1/engine/core.py:746-749). The GPU can then wait for the CPU grammar work.
• New code adds a GPU sync. A .item() or a .cpu() in the model runner path breaks the overlap without an error. Run with VLLM_GPU_SYNC_CHECK=warn to find it.
• Wasted tokens after a stop. With many short answers that end on EOS, about one row per finished request is wasted. This is normally much smaller than the gain.

Why it matters. At small batch sizes, the CPU work per step is a large part of the step time. Async scheduling hides it, and that improves both the inter-token latency and the throughput. When you change scheduling or model runner code, keep the two rules of this page. Plan with placeholders. Do not wait for the GPU before the next step is queued.