The engine client: AsyncLLM in the API server

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

The OpenAI-compatible routes do not talk to the engine core process directly. They call one object, the EngineClient, through a fixed set of methods: generate(), abort(), check_health() and some more. In vllm serve this object is AsyncLLM. It lives in the API server process, the front end.

AsyncLLM does three jobs. On the way in, it checks the rendered prompt and makes an EngineCoreRequest. In the middle, its engine core client sends the request over ZMQ and receives EngineCoreOutputs. On the way out, its OutputProcessor detokenizes the token ids and makes a RequestOutput for each request.

Many requests run at the same time, but the engine core sends one message per step for all of them. So AsyncLLM has one background task, the output handler, that reads all outputs. It puts each request's output into a small queue for that request, the RequestOutputCollector. The task that serves the HTTP request waits only on its own collector.

An everyday comparison. Think of a post office with mailboxes. A truck (the engine core) brings one bag of letters after each step. One clerk (the output handler) sorts the bag into the mailboxes. Each customer (a generate() task) waits at its own mailbox. The comparison stops in one place: a mailbox holds only one item. When a new letter arrives before the customer takes the old one, the clerk staples the two letters together.

The picture#

Read panel 2 along the arrows: the top row from left to right, then the bottom row from right to left. All boxes in panel 2 run on the one event loop of the API server process. Only the collector belongs to one request; everything else is shared by all requests.

In the code#

Step 1: the EngineClient interface#

EngineClient is an abstract class in vllm/engine/protocol.py. It declares four attributes and the state properties that the server checks:

class EngineClient(ABC):
    """Protocol class for Clients to Engine."""

    vllm_config: VllmConfig
    model_config: ModelConfig
    renderer: BaseRenderer
    input_processor: InputProcessor

    @property
    @abstractmethod
    def is_running(self) -> bool: ...

    @property
    @abstractmethod
    def is_stopped(self) -> bool: ...

    @property
    @abstractmethod
    def errored(self) -> bool: ...

    @property
    def should_keep_api_server_alive(self) -> bool:
        return False

    @property
    @abstractmethod
    def dead_error(self) -> BaseException: ...

The main abstract methods are generate (vllm/engine/protocol.py:93-115), encode for pooling models, abort (vllm/engine/protocol.py:132-141) and check_health (vllm/engine/protocol.py:163-166). Sleep, wake-up, LoRA, cache resets and pause are abstract too (vllm/engine/protocol.py:184-256). collective_rpc and the weight-transfer methods have a default that raises NotImplementedError (vllm/engine/protocol.py:275-283, vllm/engine/protocol.py:303-335). In the vllm/ package, AsyncLLM is the one class that implements it (vllm/v1/engine/async_llm.py:93).

The serving classes use the attributes too. They take the renderer and the input processor from the engine client (vllm/entrypoints/generate/base/serving.py:150-151). Before they start a response, they check errored and call check_admission (vllm/entrypoints/generate/base/serving.py:169-181).

Step 2: construction#

The API server calls AsyncLLM.from_vllm_config (vllm/entrypoints/launchers/api_server/entry.py:93-103). This class method picks the executor class from the config and turns disable_log_stats into log_stats:

    @classmethod
    def from_vllm_config(
        cls,
        vllm_config: VllmConfig,
        start_engine_loop: bool = True,
        usage_context: UsageContext = UsageContext.ENGINE_CONTEXT,
        stat_loggers: list[StatLoggerFactory] | None = None,
        enable_log_requests: bool = False,
        aggregate_engine_logging: bool = False,
        disable_log_stats: bool = False,
        client_addresses: dict[str, Any] | None = None,
        client_count: int = 1,
        client_index: int = 0,
    ) -> "AsyncLLM":
        # Create the LLMEngine.
        return cls(
            vllm_config=vllm_config,
            executor_class=Executor.get_class(vllm_config),
            start_engine_loop=start_engine_loop,
            stat_loggers=stat_loggers,
            log_requests=enable_log_requests,
            log_stats=not disable_log_stats,
            aggregate_engine_logging=aggregate_engine_logging,
            usage_context=usage_context,
            client_addresses=client_addresses,
            client_count=client_count,
            client_index=client_index,
        )

The constructor then builds the parts in this order: the renderer, the InputProcessor, the output processor and the engine core client (vllm/v1/engine/async_llm.py:167-201). With one API server, the engine core client also starts the engine core processes. In every case it waits until each engine core is ready (page The process and thread architecture of vllm serve, Step 2). Then the constructor makes the stat loggers when log_stats is on (vllm/v1/engine/async_llm.py:203-214). Stat logger plugins can turn on log_stats even when you disable the default loggers (vllm/v1/engine/async_llm.py:155-165).

At the end, the constructor starts the output handler, but only if an event loop already runs. Otherwise the first add_request starts it (vllm/v1/engine/async_llm.py:218-224, vllm/v1/engine/async_llm.py:523-526).

Step 3: generate() and add_request()#

The chat handler calls self.engine_client.generate(...) with the rendered engine input and the sampling parameters (vllm/entrypoints/openai/chat_completion/serving.py:396-408). generate() is an async generator, so this call only makes the generator object. The code runs when the HTTP stream asks for the first chunk (see Tour 1, stop 4):

        q: RequestOutputCollector | None = None
        try:
            q = await self.add_request(
                request_id,
                prompt,
                sampling_params,
                lora_request=lora_request,
                tokenization_kwargs=tokenization_kwargs,
                trace_headers=trace_headers,
                priority=priority,
                data_parallel_rank=data_parallel_rank,
                session_id=session_id,
                kv_hints=kv_hints,
                prompt_text=prompt_text,
                reasoning_ended=reasoning_ended,
                reasoning_parser_kwargs=reasoning_parser_kwargs,
            )

            # The output_handler task pushes items into the queue.
            # This task pulls from the queue and yields to caller.
            finished = False
            while not finished:
                # Note: drain queue without await if possible (avoids
                # task switching under load which helps performance).
                out = q.get_nowait() or await q.get()

                # Note: both OutputProcessor and EngineCore handle their
                # own request cleanup based on finished.
                assert isinstance(out, RequestOutput)
                finished = out.finished
                if out is not STREAM_FINISHED:
                    yield out

add_request does the work before the loop. For a request with n = 1, the important lines are:

        self.input_processor.assign_request_id(request)

        # We start the output_handler on the first call to add_request() so
        # we can call __init__ before the event loop, which enables us
        # to handle startup failure gracefully in the OpenAI server.
        self._run_output_handler()

        # Create a new output collector for the request.
        queue = RequestOutputCollector(params.output_kind, request.request_id)

        # Use cloned params that may have been updated in process_inputs()
        params = request.params

        if is_pooling or params.n == 1:
            await self._add_request(request, prompt_text, None, 0, queue)
            return queue

1. Line 521: assign_request_id keeps the caller's id as external_req_id and adds 8 random characters to request_id (vllm/v1/engine/input_processor.py:353-371).
2. Line 529: one collector per request. Its mode depends on output_kind: DELTA for a stream.
3. Line 535: _add_request registers the request in the output processor first, and then sends it (vllm/v1/engine/async_llm.py:580-584). So an output can never arrive for a request that the output processor does not know.

Step 4: the output handler and its chunks#

One task serves all requests. It loops for the whole life of the server:

        async def output_handler():
            try:
                while True:
                    # 1) Pull EngineCoreOutputs from the EngineCore.
                    outputs = await engine_core.get_output_async()
                    num_outputs = len(outputs.outputs)

                    iteration_stats = (
                        IterationStats() if (log_stats and num_outputs) else None
                    )

                    # Split outputs into chunks of at most
                    # VLLM_V1_OUTPUT_PROC_CHUNK_SIZE, so that we don't block the
                    # event loop for too long.
                    engine_core_outputs = outputs.outputs
                    for start in range(0, num_outputs, chunk_size):
                        end = start + chunk_size
                        outputs_slice = engine_core_outputs[start:end]

                        # 2) Process EngineCoreOutputs.
                        processed_outputs = output_processor.process_outputs(
                            outputs_slice,
                            outputs.timestamp,
                            iteration_stats,
                            weight_version=outputs.weight_version,
                        )
                        # NOTE: RequestOutputs are pushed to their queues.
                        assert not processed_outputs.request_outputs

                        # Allow other asyncio tasks to run between chunks
                        if end < num_outputs:
                            await asyncio.sleep(0)

                        # 3) Abort any reqs that finished due to stop strings.
                        if processed_outputs.reqs_to_abort:
                            await engine_core.abort_requests_async(
                                processed_outputs.reqs_to_abort
                            )

• Line 880 waits for the next EngineCoreOutputs from the engine core client.
• Lines 891-893 cut the list into slices of VLLM_V1_OUTPUT_PROC_CHUNK_SIZE outputs, 128 by default (vllm/envs.py:1474-1476).
• Lines 915-916 give the event loop to other tasks between two slices. The comment at lines 887-889 gives the reason: one large step must not block the event loop for too long.
• Lines 919-922 abort the requests that the detokenizer stopped with a stop string. The engine core cannot know about stop strings, because it has no text.

After all slices, the handler updates the LoRA state and records the scheduler and iteration stats (vllm/v1/engine/async_llm.py:924-935).

Step 5: the collector hands the output over#

    def __init__(self, output_kind: RequestOutputKind, request_id: str):
        self.aggregate = output_kind == RequestOutputKind.DELTA
        self.request_id = request_id
        self.output: RequestOutput | PoolingRequestOutput | Exception | None = None
        self.ready = asyncio.Event()

        self._input_stream_task: asyncio.Task | None = None

    def put(self, output: RequestOutput | PoolingRequestOutput | Exception) -> None:
        """Non-blocking put operation."""
        if self.output is None or isinstance(output, Exception):
            self.output = output
            self.ready.set()
        elif isinstance(self.output, RequestOutput) and isinstance(
            output, RequestOutput
        ):
            # This ensures that request outputs with different request indexes
            # (if n > 1) do not override each other.
            self.output.add(output, aggregate=self.aggregate)
        elif isinstance(self.output, PoolingRequestOutput) and isinstance(
            output, PoolingRequestOutput
        ):
            self.output = output

    async def get(self) -> RequestOutput | PoolingRequestOutput:
        """Get operation blocks on put event."""
        while (output := self.output) is None:
            await self.ready.wait()
        self.output = None
        self.ready.clear()
        if isinstance(output, Exception):
            raise output
        return output

The collector has one slot (self.output) and an asyncio.Event. When the slot is full and a new RequestOutput comes, put() merges it into the old one (line 79). With DELTA, the merge appends the new text and token ids (vllm/outputs.py:220-225). So a slow HTTP writer gets one larger chunk, and no token is lost. An exception in the slot replaces any output, and get() raises it (lines 71 and 91-92).

Step 6: abort when the client goes away#

When the HTTP client disconnects, Starlette cancels the response task. The cancel reaches generate() as CancelledError or GeneratorExit:

        except (asyncio.CancelledError, GeneratorExit):
            if q is not None:
                await self.abort(q.request_id, internal=True)
            if self.log_requests:
                logger.info("Request %s aborted.", request_id)
            raise

    async def abort(
        self, request_id: str | Iterable[str], internal: bool = False
    ) -> None:
        """Abort RequestId in OutputProcessor and EngineCore."""
        request_ids = (
            (request_id,) if isinstance(request_id, str) else as_list(request_id)
        )
        all_request_ids = self.output_processor.abort_requests(request_ids, internal)
        await self.engine_core.abort_requests_async(all_request_ids)

abort() first removes the request from the output processor. It returns the internal ids, which include the children of an n > 1 request (vllm/v1/engine/output_processor.py:599-661). Then it sends one ABORT message with these ids to the engine core. Before a streaming response starts, the route decorator with_cancellation watches for the disconnect itself (vllm/entrypoints/serve/utils/api_utils.py:85-109).

Step 7: errors and engine death#

vLLM has two engine exceptions. EngineGenerateError is for one request and "Recoverable". EngineDeadError means that the engine core died, and it is "Unrecoverable" (vllm/v1/engine/exceptions.py:6-22). The death becomes visible in four places:

1. The engine core sends ENGINE_CORE_DEAD. The client sets engine_dead and raises EngineDeadError (vllm/v1/engine/core_client.py:581-584). The monitor thread sets the same flag when an engine core process exits (vllm/v1/engine/core_client.py:829-838).
2. The exception reaches the output handler, which puts it into every collector (vllm/v1/engine/async_llm.py:936-944, vllm/v1/engine/output_processor.py:593-597). Each waiting generate() raises it again, without an abort (vllm/v1/engine/async_llm.py:814-818).
3. errored becomes true when the flag is set or the output handler task has ended (vllm/v1/engine/async_llm.py:1373-1384). New requests then fail at once (vllm/v1/engine/async_llm.py:418-419), and /health returns 503 (vllm/entrypoints/serve/instrumentator/health.py:29-33).
4. A watchdog task checks every 5 seconds and stops uvicorn, unless VLLM_KEEP_ALIVE_ON_ENGINE_DEATH is set (vllm/entrypoints/launchers/launcher.py:314-337).

Step 8: which engine core client#

AsyncLLM always uses a multiprocess client: make_async_mp_client chooses the class by the data parallel settings:

        if parallel_config.data_parallel_size > 1:
            if parallel_config.data_parallel_external_lb:
                # External load balancer - client per DP rank.
                return DPAsyncMPClient(
                    *client_args,
                    renderer=renderer,
                )
            # Internal load balancer - client balances to all DP ranks.
            return DPLBAsyncMPClient(
                *client_args,
                renderer=renderer,
            )
        return AsyncMPClient(
            *client_args,
            renderer=renderer,
        )

• AsyncMPClient (DP = 1): one engine core, one ZMQ identity.
• DPLBAsyncMPClient (DP > 1, internal load balancing): picks an engine core for each request by load (vllm/v1/engine/core_client.py:1743-1804).
• DPAsyncMPClient (DP > 1, external load balancer): sends everything to its own engine core (vllm/v1/engine/core_client.py:1611-1612).

The page v1/engine/core_client.py: the engine core clients reads these classes line by line. The page Data parallel serving: several engine cores behind one endpoint explains the load balancing modes.

Step 9: sleep, wake-up and collective RPCs at a glance#

Control methods do not go through generate(). AsyncLLM forwards them as utility calls. For example, sleep() clears the multimodal cache and calls engine_core.sleep_async (vllm/v1/engine/async_llm.py:1196-1202). collective_rpc forwards any worker method name (vllm/v1/engine/async_llm.py:1247-1257). In the engine core, the utility method collective_rpc calls the executor, which runs the method on every worker (vllm/v1/engine/core.py:1098-1105).

The HTTP routes /sleep, /wake_up and /collective_rpc exist only when VLLM_SERVER_DEV_MODE is set (vllm/entrypoints/launchers/api_server/routers.py:46-49, vllm/entrypoints/serve/__init__.py:43-65).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

        """Add new request to the AsyncLLM."""
        if self.errored:
            raise EngineDeadError()

        if trace_headers and self.observability_config.otlp_traces_endpoint is None:
            log_tracing_disabled_warning()
            trace_headers = None

        is_pooling = isinstance(params, PoolingParams)

        if (
            self.vllm_config.cache_config.kv_sharing_fast_prefill
            and not is_pooling
            and params.prompt_logprobs
        ):
            raise VLLMValidationError(
                "--kv-sharing-fast-prefill produces incorrect logprobs for "
                "prompt tokens, please disable it when the requests need "
                "prompt logprobs"
            )

        else:
            if isinstance(prompt, dict) and "type" in prompt:
                # Rendered EngineInput; no blocking preprocessing needed.
                request = self.input_processor.process_inputs(
                    request_id,
                    prompt,
                    params,
                    supported_tasks=await self.get_supported_tasks(),
                    arrival_time=arrival_time,
                    lora_request=lora_request,
                    tokenization_kwargs=tokenization_kwargs,
                    trace_headers=trace_headers,
                    priority=priority,
                    data_parallel_rank=data_parallel_rank,
                    session_id=session_id,
                    kv_hints=kv_hints,
                )
            else:
                # Raw prompts require tokenization and possibly multimodal
                # processing, which must not block the event loop.
                request = await self.input_processor.process_inputs_async(
                    request_id,
                    prompt,
                    params,
                    supported_tasks=await self.get_supported_tasks(),
                    arrival_time=arrival_time,
                    lora_request=lora_request,
                    tokenization_kwargs=tokenization_kwargs,
                    trace_headers=trace_headers,
                    priority=priority,
                    data_parallel_rank=data_parallel_rank,
                    session_id=session_id,
                    kv_hints=kv_hints,
                )
            prompt_text, _, _ = extract_prompt_components(self.model_config, prompt)

        parent_params = params
        assert isinstance(parent_params, SamplingParams)

        # Fan out child requests (for n>1).
        parent_request = ParentRequest(request)
        child_requests: list[EngineCoreRequest] = []
        for idx in range(parent_params.n):
            request_id, child_params = parent_request.get_child_info(idx)
            child_request = request if idx == parent_params.n - 1 else copy(request)
            child_request.request_id = request_id
            child_request.sampling_params = child_params
            child_requests.append(child_request)

        self.check_admission(parent_params.n, parent_request.request_id)
        try:
            for idx, child_request in enumerate(child_requests):
                self.output_processor.add_request(
                    child_request, prompt_text, parent_request, idx, queue
                )

            for child_request in child_requests:
                await self.engine_core.add_request_async(child_request)
                if self.log_requests:
                    logger.info("Added request %s.", child_request.request_id)
        except BaseException:
            await self.abort(parent_request.request_id, internal=True)
            raise
        return queue

What can go wrong#

• The server exits after an engine error. This is the default for EngineDeadError: the watchdog stops uvicorn (vllm/entrypoints/launchers/launcher.py:325-337). Read the log of the engine core process for the root cause. The message of the exception says "See stack trace (above) for the root cause" (vllm/v1/engine/exceptions.py:19-21).
• A stop string ends the text, but the engine still runs a step. The engine core does not see stop strings. The output handler sends an ABORT after the detokenizer finds one (vllm/v1/engine/async_llm.py:918-922). The default loop schedules the next step before it processes the output of the current one (vllm/v1/engine/core.py:724-759, vllm/v1/engine/core.py:767-785). So the next step for that request is usually already on the GPU, and more steps can start before the ABORT arrives. (Derived.) The front end ignores these outputs, because the request state is gone (vllm/v1/engine/output_processor.py:759-762).
• The event loop is busy. All requests share one event loop and one output handler. A very large batch, long stop strings or many logprobs make process_outputs slow, and all streams wait. More API servers (--api-server-count) split this CPU work (The process and thread architecture of vllm serve).
• The 503 errors come early. With max_num_queued_tokens set, the admission check counts full prompt lengths, so it can reject requests while the GPU still has room (vllm/v1/engine/async_llm.py:337-341).

Why it matters. Every request of vllm serve goes through AsyncLLM twice: once on the way in and once on the way out. Its single event loop and its error handling decide how the server behaves under load and after a crash.