What vLLM is today

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

What vLLM is#

vLLM calls itself "a fast and easy-to-use library for LLM inference and serving" (README.md:24). The package description says "a high-throughput and memory-efficient inference and serving engine for LLMs" (pyproject.toml:22). An inference engine runs a trained model and makes its tokens. It adds batching, a KV cache and a scheduler around the model (What an LLM does↗). A serving engine also accepts requests from many users over the network and streams the answers back.

This site explains vLLM as a serving engine. The main path is the command vllm serve. It starts an OpenAI-compatible server: "Launch a local OpenAI-compatible API server to serve LLM completions via HTTP" (vllm/entrypoints/cli/serve.py:39-40). Any client that can talk to the OpenAI API can then talk to your model.

One engine: V1#

vLLM has exactly one engine at this commit: the V1 engine in vllm/v1/. The older V0 engine is gone. Only two old names remain, and both are aliases of V1 classes:

• vllm.engine.async_llm_engine.AsyncLLMEngine = AsyncLLM (vllm/engine/async_llm_engine.py:4-6).
• vllm.engine.llm_engine.LLMEngine = V1LLMEngine (vllm/engine/llm_engine.py:4-6).

So when you read "AsyncLLMEngine" in old examples, it is the V1 AsyncLLM. This site uses only the V1 names.

Two ways to use it#

1. Online serving with vllm serve (this site). An HTTP client sends a request, for example POST /v1/chat/completions. The API server turns it into an engine request and streams the tokens back. The page The OpenAI-compatible API explains the API.
2. Offline inference with the LLM class (a side note). Your Python script makes LLM(model=...) and calls generate() or chat(). There is no HTTP server. The class builds a V1 LLMEngine (vllm/entrypoints/llm.py:341-343). Its own docstring says that it "is intended to be used for offline inference" (vllm/entrypoints/llm.py:171-173).

What runs where#

One vllm serve deployment is a group of processes. Each one has its own job:

• The front end (the API server process) runs the HTTP routes, the chat template, the tokenizer and AsyncLLM.
• The EngineCore process runs the scheduler and the KV cache manager in a loop. It tells the workers what to compute in each engine step.
• The workers own the GPUs and run the model. With one GPU, the worker is an object inside the engine core process.

The page The process and thread architecture of vllm serve explains this split in full. The page Processes, ZMQ and shared memory explains how the processes talk to each other.

An everyday comparison#

Think of a restaurant. The waiter (the front end) takes orders from many tables and brings the dishes back. The head chef (the engine core) decides which dishes to cook in the next round. The cooks at the stoves (the workers) do the cooking. A take-away counter (the LLM class) uses the same kitchen, but without waiters.

The comparison stops here: a real kitchen cooks each dish once. Without speculative decoding, vLLM makes at most one new token per request in each step, and repeats the step until each answer is complete.

The picture#

Read panel 1 from left to right: this is the path of every request on this site. Panel 2 shows the offline path. It shares the engine core, but the front end is your own script. The blue, amber and violet colours mark the online front end, the offline path and the shared engine core.

In vLLM#

This is a basics page, so this part gives only pointers. The tour Tour: Life of a chat request follows one request through all of the code.

1. The command. The package installs one console script, vllm, that calls vllm.entrypoints.cli.main:main (pyproject.toml:42-43). With one API server, vllm serve runs the server in the same process as the command (vllm/entrypoints/cli/serve.py:154-156).
2. The front end. The API server makes an AsyncLLM (vllm/entrypoints/launchers/api_server/entry.py:94-99). AsyncLLM always starts the engine core in a separate process, through make_async_mp_client (vllm/v1/engine/async_llm.py:193-201). The page The engine client: AsyncLLM in the API server explains it.
3. The engine core. EngineCoreProc in vllm/v1/engine/core.py runs the busy loop. The page The engine core loop: threads, queues and the busy loop explains it.

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

from vllm.v1.engine.async_llm import AsyncLLM

AsyncLLMEngine = AsyncLLM  # type: ignore
"""The `AsyncLLMEngine` class is an alias of [vllm.v1.engine.async_llm.AsyncLLM][]."""

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

    def make_client(
        multiprocess_mode: bool,
        asyncio_mode: bool,
        vllm_config: VllmConfig,
        executor_class: type[Executor],
        log_stats: bool,
        renderer: BaseRenderer | None = None,
    ) -> "EngineCoreClient":
        # renderer is passed through to the multiprocess clients, which start
        # the frontend MM warmup (renderer.start_mm_warmup_in_background) once
        # the engine-core processes have been forked, so warmup overlaps
        # engine-core load without a live thread at fork() time. In-process
        # clients do not take a renderer: the engine core is built in this
        # process, so there is no other process to overlap with and the MM
        # warmup runs synchronously inside renderer.warmup() instead.
        # TODO: support this for debugging purposes.
        if asyncio_mode and not multiprocess_mode:
            raise NotImplementedError(
                "Running EngineCore in asyncio without multiprocessing "
                "is not currently supported."
            )

        if multiprocess_mode and asyncio_mode:
            return EngineCoreClient.make_async_mp_client(
                vllm_config,
                executor_class,
                log_stats,
                renderer=renderer,
            )

        if multiprocess_mode and not asyncio_mode:
            return SyncMPClient(
                vllm_config,
                executor_class,
                log_stats,
                renderer=renderer,
            )

        return InprocClient(vllm_config, executor_class, log_stats)

What can go wrong#

• Old tutorials. A tutorial that says "V0", --use-v2-block-manager or AsyncLLMEngine internals describes code that is gone. The two old class names still import, but they are V1 classes (vllm/engine/async_llm_engine.py:6, vllm/engine/llm_engine.py:6).
• Offline and online defaults differ. The LLM class turns statistics off by default (vllm/entrypoints/llm.py:225-226). It also gets other scheduler defaults than vllm serve, because the usage context differs (vllm/entrypoints/llm.py:341-343, vllm/engine/arg_utils.py:2960-2993). A benchmark with LLM does not tell you how vllm serve behaves.
• No platform found. When no detector is active, vLLM uses UnspecifiedPlatform (vllm/platforms/__init__.py:290-292), and later code fails. Check that the right GPU drivers or the platform plugin package are installed.
• Two platform plugins. Two installed and active platform plugins stop start-up with RuntimeError (vllm/platforms/__init__.py:272-276).
• A version string of dev. If vllm/_version.py is missing (for example, a source tree that was never built), the version is "dev" with a warning (vllm/version.py:6-12).

Why it matters. Know your entry point and its processes. Then you know which pages of this site apply to you, and which process owns each flag and each log line.