vLLM Explained

vLLM Explained

Code version: vLLM @ 3e44c0d · 2026-10-22

vLLM is "a fast and easy-to-use library for LLM inference and serving" (README.md:24). Its main use is online serving: the command vllm serve starts an OpenAI-compatible server that answers HTTP requests (vllm/entrypoints/cli/serve.py:39-40). Behind the HTTP routes, a scheduler runs one engine step after the other and batches the requests of many clients on the GPUs. Outside code can change much of this through plugins.

Who this site is for. You know Python. You read DeepInfer Explained, or you know the LLM basics that it teaches. Now you want to become proficient in online serving with the latest vLLM: what happens to a request inside vllm serve, how to configure, tune, monitor and scale a deployment, and how plugins extend vLLM. This site does not teach the basics again. When a page needs one, it gives a one-line recap and a link to DeepInfer Explained. The site explains the code at commit 3e44c0d. To get exactly this version of the code, read The code version this site explains.

What it runs"200+ model architectures on Hugging Face": decoder-only and Mixture-of-Experts LLMs, hybrid and state-space models, multimodal models, embedding and reward models (README.md:53-60). The default model of vllm serve is Qwen/Qwen3-0.6B (vllm/config/model.py:155, vllm/engine/arg_utils.py:486, vllm/entrypoints/cli/serve.py:39-40).
HardwareBuilt-in platform detectors: TPU, CUDA (NVIDIA), ROCm (AMD), XPU (Intel) and CPU (vllm/platforms/__init__.py:231-237). The TPU platform class comes from the separate package tpu_inference (vllm/platforms/tpu.py:9-19). Other hardware comes as platform plugins, for example Intel Gaudi, IBM Spyre and Huawei Ascend (README.md:51).
Code size8,357 files. Python: 5,471 files, about 2.0 million lines; the package vllm/ has 2,907 Python files and about 1.13 million lines; the Python files in tests/ have about 0.77 million lines. C++ and CUDA: 313 files, about 143,000 lines (kernels, not explained here). Rust: 375 files, about 152,000 lines (an experimental front end). Method: the files of git ls-files at the pinned commit, grouped by extension (.py; .cpp .cu .cuh .h .hpp; .rs); a line is a newline, plus a last line without one. See Code map: the folder tree.
Run itvllm serve <model>, for example vllm serve Qwen/Qwen3-0.6B. The command vllm is a console script (pyproject.toml:42-43). The server listens on port 8000 by default (vllm/entrypoints/common/cli_args.py:286).

The system map#

The map shows the path of one streaming chat request through vllm serve, and the parts that every step uses. Each box is a link. Click a box to open its page. Grey links are pages that are not written yet. The coloured stripe on the left of a box shows the process that runs the code.

The path of one chat request

API server process

1 · The API server gets POST /v1/chat/completions

vllm/entrypoints/openai/chat_completion/api_router.py · serving.py

A FastAPI route gives the request to OpenAIServingChat. It checks the model name and calls the renderer (box 2). Then it makes the SamplingParams.

Code: the route · handler · check, render · sampling params · Also: the OpenAI API · Tour 1

a ChatCompletionRequest

API server process

2 · The renderer applies the chat template

vllm/renderers/ (online_renderer.py · base.py · hf.py)

Turns the chat messages into one prompt with the model's chat template, then into token ids. It also sets the arrival time of the request.

Code: render_chat_async · Also: checkpoint files

token ids, then SamplingParams from OpenAIServingChat

API server process

3 · AsyncLLM adds the request

vllm/v1/engine/async_llm.py · input_processor.py

The input processor makes an EngineCoreRequest. AsyncLLM registers a queue for the outputs of this request, then sends it to the engine core.

Code: process_inputs · queue and send · Also: async_llm.py · data contracts

EngineCoreRequest

From the API server to the engine core

4 · ZMQ and msgpack carry the request

vllm/v1/engine/core_client.py → vllm/v1/engine/core.py

The engine core client encodes the request with msgpack and sends it on a ZMQ ROUTER socket. An input thread in the engine core process receives it.

Code: send · msgpack · ROUTER · input thread · Also: processes and IPC

the request in the waiting queue

Engine core process

5 · EngineCore runs a step: the scheduler picks the work

vllm/v1/engine/core.py · vllm/v1/core/sched/scheduler.py

Each step schedules, executes the model and updates the requests. By default, async scheduling is on, and the loop runs step_with_batch_queue. The scheduler gives each request a number of tokens from the token budget and KV cache blocks for them.

Code: step_with_batch_queue · step_fn · async default · batch queue size · Also: the core loop · async scheduling · one step

SchedulerOutput: a direct call with one GPU; a shared-memory message queue to every worker process with more GPUs

Worker: engine core process (1 GPU) or one process per GPU

6 · The worker and Model Runner V2 run the model

vllm/v1/executor/uniproc_executor.py · multiproc_executor.py · vllm/v1/worker/gpu_worker.py · vllm/v1/worker/gpu/model_runner.py

The model runner builds the input tensors and the attention metadata. Then it runs the forward pass. At the default level O2, a decode batch with 1 token per request replays a FULL CUDA graph, and another batch uses PIECEWISE graphs. A batch larger than the largest captured size runs without a graph. The attention backend can lower the mode at start-up.

Code: execute_model · graph or eager · O2 mode · dispatch · lower mode · Also: executors · attention backends · one forward pass

logits

Worker: same process as box 6

7 · The sampler picks the next token

vllm/v1/worker/gpu/sample/sampler.py

Picks one token per request from the logits. With speculative decoding, a rejection sampler checks the draft tokens.

Code: sampler call · Also: speculative decoding · structured outputs

sampled token ids → update_from_output in the engine core → EngineCoreOutputs over ZMQ to the API server

API server process

8 · The output processor and the SSE stream

vllm/v1/engine/output_processor.py · chat_completion/serving.py

An output task in AsyncLLM detokenizes the new tokens and fills the queue of each request. The stream generator writes SSE chunks, and at the end data: [DONE].

Code: output_handler · process_outputs · [DONE] · Also: tool and reasoning parsers

Around every step

Every process

Configuration and environment variables

vllm/config/ · vllm/engine/arg_utils.py · vllm/envs.py

The CLI flags become one VllmConfig. Environment variables VLLM_* change behaviour in every process.

Code: VllmConfig · envs · Also: flags · env vars

Every process

Plugins and current_platform

vllm/plugins/ · vllm/platforms/

Each process loads the plugin groups that it needs. current_platform is the one platform object of a process.

Code: load_general_plugins · Also: plugin map · platforms · plugins/__init__.py

Engine core process

KV cache manager and prefix caching

vllm/v1/core/kv_cache_manager.py · block_pool.py

The scheduler owns it. It gives blocks to requests and reuses the cached blocks of a shared prompt prefix. Prefix caching is on by default for generative decoder-only and hybrid models.

Code: made by the scheduler · default · which models · Also: prefix caching · hybrid KV cache

API server process

Metrics and logging

vllm/v1/metrics/ · /metrics route

Stat loggers in AsyncLLM record the request metrics. The API server shows them in Prometheus format at /metrics.

Code: stat loggers · /metrics · Also: serving metrics

Worker (same process as box 6)

torch.compile and CUDA graphs

vllm/compilation/ · vllm/v1/worker/gpu/model_runner.py

At the default level O2, the worker compiles the model with torch.compile and captures FULL and PIECEWISE CUDA graphs at start-up. A later step replays a graph when the batch fits a captured size. --enforce-eager turns both off.

Code: O2 default · compile mode · graph mode · enforce eager · capture_model · Also: tuning · start-up tour

API server process (front end)engine core processworker (in the engine core process with one GPU; one process per GPU with more)crosses a process boundaryevery process

How to read the map. Follow the left column from 1 to 8. Boxes 1 to 3 and 8 run in the API server process, the front end. Box 5 runs in the EngineCore process. Boxes 6 and 7 run in a worker, which owns one GPU. With one GPU, the worker runs inside the engine core process. With more GPUs, each worker is a separate process. Box 4 and the end of the arrow after box 7 cross a process boundary. The arrow after box 5 crosses one only when there are worker processes. The right column shows parts that every step uses. The page Tour: Life of a chat request walks the same path in 16 stops.

Start here#

Choose the path that matches your goal. Each path is a list of pages in reading order.

How to use this site#

  • Levels. Each page has a badge: Beginner, Intermediate or Advanced. A "Before you read" box lists the pages to read first.
  • Two depths on each page. A page starts with one sentence, a plain explanation and a diagram. Then it shows the real code. Deep dive boxes give the full detail: types, shapes, edge cases and environment variables.
  • Check yourself. Most pages end with 3 to 5 questions. Answer each one, then click "Show answer".
  • Search. Press /, then type a word, a file name or an environment variable.
  • Glossary. Words with a dotted line, such as EngineCore, are glossary terms. Point to them, or tap them on a phone, to see a short definition. Glossary lists all terms.
  • Code. The build copies every code excerpt from the repository. Every citation opens the file on GitHub at commit 3e44c0d.
  • DeepInfer links. A link with a small "↗" opens DeepInfer Explained or GitHub in a new tab.
  • Grey entries in the menu are pages that are not written yet. Diagrams open at full size when you click them.

What this site does not explain#

  • Kernel internals. The site shows how vLLM selects and calls a kernel (a device program), but not how the kernel works inside: not csrc/ (csrc/), and not the Triton, CUDA, CuTe or Helion kernel code.
  • Legacy code. V0 is gone; only aliases remain, for example AsyncLLMEngine = AsyncLLM (vllm/engine/async_llm_engine.py:6). Model Runner V2 is the default. Model Runner V1 (vllm/v1/worker/gpu_model_runner.py) gets one note on The model runner (Model Runner V2).
  • Outside libraries such as PyTorch, FlashAttention, FlashInfer, NCCL, Ray, ZMQ and FastAPI.
  • One paragraph only (on What vLLM is today): the experimental Rust front end (rust/) and the per-family model packages in vllm/models/ (vllm/models/). The special code of single model families is also out of scope; llama.py is the one model example.

The code version this site explains#

This site explains one exact commit of the vLLM repository. The repository has 23,594 commits at this point. It has no git submodules, so a clone and a checkout are enough. Use this version when you study the code with the site.

To get exactly this version of the code, run these commands:

Commit3e44c0db26f3530d674d255dafabaedcf0e3d742 (short: 3e44c0d)
Date2026-10-22 22:21:12 +0800
Subject“[Bugfix][Benchmark] Support boolean sweep plot filters (#60774)”
Browsevllm-project/vllm at 3e44c0d
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout 3e44c0db26f3530d674d255dafabaedcf0e3d742