Glossary

Referenceabout 71 min read

Code version: vLLM @ 3e44c0d · 2026-10-22

This page lists every term that has a tooltip on this site. Each entry gives a short definition. Many entries link to a page with the full explanation.

# A B C D E F G H I J K L M N O P Q R S T U V W Z

##

--goodput also: goodput SLO, request goodput

The vllm bench serve flag that sets SLOs as KEY:VALUE in milliseconds (ttft, tpot, e2el). A request is good when it meets every given SLO.

Flag vllm/benchmarks/serve.py:1867-1878; check vllm/benchmarks/serve.py:669-692.

--kv-cache-memory-bytes also: kv_cache_memory_bytes, --kv-cache-memory

A fixed KV cache size per GPU, in bytes (1G = 1 GiB). With it, the worker still runs the profile run, but skips the memory measurement and ignores --gpu-memory-utilization.

Field vllm/config/cache.py:371-378; worker path vllm/v1/worker/gpu_worker.py:653-679. The start-up log suggests a value but names it --kv-cache-memory (vllm/v1/worker/gpu_worker.py:1046-1053).

--performance-mode also: performance_mode, throughput mode, interactivity mode

A vllm serve flag with the values balanced (default), interactivity and throughput. Throughput doubles the default token budget and max_num_seqs; interactivity captures CUDA graphs for every batch size from 1 to 32.

Field vllm/config/vllm.py:440-445; throughput vllm/engine/arg_utils.py:3171-3175; interactivity vllm/config/vllm.py:2552-2556.

@config decorator also: @config, config decorator

The decorator in vllm/config/utils.py that makes a config class a pydantic dataclass whose settings forbid unknown fields.

Defined at vllm/config/utils.py:51-82; extra="forbid" at vllm/config/utils.py:71.

A#

acceptance rate also: mean acceptance length, draft acceptance rate

Accepted draft tokens divided by proposed draft tokens. The mean acceptance length is 1 + accepted tokens per verification round.

PromQL in the docstring of SpecDecodingProm (vllm/v1/spec_decode/metrics.py:177-196).

add_generation_prompt also: continue_final_message, generation prompt

A chat template variable (default true) that adds the start of a new assistant turn at the end of the prompt, so the model answers next.

Request field at vllm/entrypoints/openai/chat_completion/protocol.py:324-331; it cannot be true together with continue_final_message (vllm/entrypoints/openai/chat_completion/protocol.py:1045-1055).

all-reduce also: allreduce, all reduce

A collective operation: every GPU in a group gives one tensor, and every GPU gets back the element-wise sum of all of them.

all2all backend also: --all2all-backend, all-to-all backend

The communication method that moves MoE tokens to the ranks of their experts and back. Set with --all2all-backend; the default is allgather_reducescatter.

Choices and default at vllm/config/parallel.py:45-60, vllm/config/parallel.py:220-234.

allocate_slots also: KVCacheManager.allocate_slots

The KVCacheManager method that the scheduler calls for each scheduled request. It adds prefix-cache hits, gets new blocks for the new tokens, caches full blocks, or returns None when blocks are short.

vllm/v1/core/kv_cache_manager.py:371-618

API server also: API servers, API server process, --api-server-count

The uvicorn + FastAPI process of vllm serve that answers HTTP requests. One by default; several with --api-server-count or data parallelism.

With one API server it runs in the CLI process (vllm/entrypoints/cli/serve.py:153-156); with more, the CLI process spawns ApiServer_i processes (vllm/v1/utils.py:248-254).

app state also: app.state, init_app_state

The FastAPI app.state object of the API server. init_app_state stores the engine client, the renderer and one object per serving class in it.

Filled in vllm/entrypoints/launchers/api_server/app_state.py:24-176; route functions read it, for example request.app.state.openai_serving_chat.

async scheduling also: asynchronous scheduling, AsyncScheduler

The engine core schedules and starts step k+1 before it waits for the result of step k. So CPU work overlaps with GPU work. It is on by default.

Default: vllm/config/vllm.py:1664-1698. Batch queue: vllm/v1/engine/core.py:698-812.

AsyncLLM also: AsyncLLMEngine

The engine object in the front end of vllm serve. It turns requests into engine inputs, sends them to the engine core and streams the outputs back.

attention backend also: attention backends, AttentionBackend

A set of classes that implement attention on a platform, for example FlashAttention or FlashInfer. vLLM selects one per layer type.

attention metadata also: attn_metadata, AttentionMetadata

The per-step object that an attention backend needs to run its kernel for a group of layers: query offsets, sequence lengths, block table, slot mapping and backend-specific fields.

A metadata builder makes it from CommonAttentionMetadata. Model Runner V2 puts it into a dict keyed by layer name (vllm/v1/worker/gpu/attn_utils.py:579-580).

AttentionBackendEnum also: attention backend enum, register_backend, --attention-backend

The enum that names every attention backend. Each value is the default dotted class path; register_backend can override it or add a member.

Defined at vllm/v1/attention/backends/registry.py:34-166 (49 members). Overrides live in _ATTN_OVERRIDES (vllm/v1/attention/backends/registry.py:344).

AttentionCGSupport also: CUDA graph support, cudagraph support, _cudagraph_support

How much CUDA graph capture a metadata builder allows: ALWAYS, UNIFORM_BATCH, UNIFORM_SINGLE_TOKEN_DECODE or NEVER. The weakest level in the model can lower the CUDA graph mode.

Enum at vllm/v1/attention/backend.py:557-569; the runner takes the minimum over all groups (vllm/v1/worker/gpu/attn_utils.py:314-341).

AutoWeightsLoader also: auto weights loader

A helper that walks the module tree with the (name, tensor) stream. It calls a child's own load_weights if it has one, else the weight_loader of each parameter.

vllm/model_executor/models/utils.py:216-478.

B#

backend priority list also: priority list, _get_backend_priorities

The ordered list of attention backends that a platform tries for one kind of layer. The first backend that passes validate_configuration wins.

CUDA builds the list in _get_backend_priorities (vllm/platforms/cuda.py:84-181), keyed by MLA or not and the compute capability.

batch also: batches, batching

The set of tokens from many requests that the model runs in one engine step. The token budget limits its size.

batch queue also: max_concurrent_batches

A short queue of started engine steps in the engine core. Its size is max_concurrent_batches: 2 on one GPU with async scheduling and Model Runner V2.

vllm/v1/engine/core.py:205-211, vllm/config/vllm.py:587-597

BatchDescriptor also: batch descriptor, batch_descriptor, BatchExecutionDescriptor

A small frozen dataclass (padded token count, request count, LoRA flags) in the forward context. Piecewise CUDA graph wrappers use it as the key of their graphs.

Defined at vllm/forward_context.py:29-57. Model Runner V2 also has its own BatchExecutionDescriptor for the dispatch decision (vllm/v1/worker/gpu/cudagraph_utils.py:70-85).

block hash also: BlockHash, BlockHashWithGroupId, block_hashes

The bytes that name one full KV cache block for prefix caching: a hash of the parent block hash, the block's token ids and its extra keys. In the map it gets a 4-byte group id.

BlockHash and BlockHashWithGroupId (vllm/v1/core/kv_cache_utils.py:44-69); computed by hash_block_tokens (vllm/v1/core/kv_cache_utils.py:707-737).

block pool also: BlockPool

The scheduler's record of all KV cache blocks: which blocks are free, which a request holds, and which cached blocks a prefix can reuse. It is CPU bookkeeping in the engine core; the KV tensors live in the workers.

block table also: block tables, block_ids

For each request, the list of physical block ids that hold its KV values, in token order. The scheduler sends new block ids; Model Runner V2 keeps the table on the GPU.

Scheduler side: req_to_blocks (vllm/v1/core/single_type_kv_cache_manager.py:126). Worker side: BlockTables (vllm/v1/worker/gpu/block_table.py:17).

breakable CUDA graph also: VLLM_USE_BREAKABLE_CUDAGRAPH, eager_break_during_capture, BreakableCUDAGraphWrapper

An experimental way to get piecewise CUDA graphs without torch.compile: capture stops at each attention op, runs it eagerly, and starts a new graph segment.

Module docstring vllm/compilation/breakable_cudagraph.py:3-20; on by default only for some architectures (vllm/config/vllm.py:75-128) or with VLLM_USE_BREAKABLE_CUDAGRAPH=1.

burstiness also: --burstiness, gamma arrivals

The shape factor of the gamma distribution that vllm bench serve draws request gaps from. 1 gives a Poisson process; below 1 gives bursts; above 1 gives more even gaps.

Flag vllm/benchmarks/serve.py:1754-1765; gamma(shape=burstiness, scale=1/(rate × burstiness)) vllm/benchmarks/serve.py:504-508.

busy loop also: run_busy_loop

The main loop of the engine core process. It takes new requests from the input queue, runs one engine step, and puts the outputs into the output queue.

EngineCoreProc.run_busy_loop (vllm/v1/engine/core.py:1609-1620).

C#

cache hit cap also: max_cache_hit_length, num_tokens - 1

A prefix-cache hit covers at most num_tokens - 1 tokens, because the model must compute the last prompt token to get its logits. With whole blocks, the last block is often computed again.

vllm/v1/core/kv_cache_manager.py:289-300

cache salt also: cache_salt, salt

An optional secret string in a request. vLLM adds it to the hash of the first KV cache block, so only requests with the same salt share cached blocks.

Request field cache_salt (vllm/entrypoints/openai/chat_completion/protocol.py:499-511); used as an extra key of block 0 only (vllm/v1/core/kv_cache_utils.py:659-663).

Read more: KV connectors

console script also: console scripts, console_scripts, [project.scripts]

An entry point in the group console_scripts. The installer makes a small program with that name, for example vllm, that calls the given function.

Read more: Python entry points

create_engine_config() also: create_engine_config

The EngineArgs method that builds every sub-config in a fixed order, fills the None defaults, and returns one VllmConfig.

vllm/engine/arg_utils.py:2234-2905.

CUDA graph also: CUDA graphs, cudagraph

A recorded sequence of GPU kernel launches that the GPU can replay with one launch. It removes CPU launch overhead in small steps.

custom all-reduce also: CustomAllreduce, custom allreduce

vLLM's own all-reduce kernel for small tensors in a TP group, normally on one node. It writes through peer-to-peer GPU memory instead of NCCL.

Conditions at vllm/distributed/device_communicators/custom_all_reduce.py:214-291; size rule vllm/distributed/device_communicators/custom_all_reduce.py:499-513.

CustomOp also: custom op, custom ops

A vLLM layer class with one forward method per platform (forward_cuda, forward_native, ...). It selects one of them when it is made.

D#

data parallel rank also: DP rank

The index of one data parallel replica. In vllm serve each DP rank is one engine core process with its own workers and KV cache.

Engine choice per rank at vllm/v1/engine/core.py:1534-1543.

data parallelism also: DP, data parallel

Several copies of the model serve different requests at the same time. In vLLM, each data parallel rank has its own engine core (DP).

DBO (dual batch overlap) also: dual batch overlap, microbatching, --enable-dbo

Splits a batch into two microbatches that run in two threads, so the MoE dispatch and combine of one overlap with the compute of the other.

docs/design/dbo.md:5-11; flag vllm/config/parallel.py:258-259.

decode also: decoding

The work that makes the output tokens after the first one, usually one token per request per step. It reads the whole KV cache of the request.

device communicator also: CudaCommunicator, DeviceCommunicatorBase

The platform object that runs the collectives of a group with more than one rank. On CUDA it is CudaCommunicator, which picks pynccl, custom all-reduce, symmetric memory or FlashInfer.

Made at vllm/distributed/parallel_state.py:549-560; CUDA class vllm/platforms/cuda.py:581-584.

dispatch key also: dispatch_key, PrivateUse1

The PyTorch dispatcher key of a device, for example CUDA or PrivateUse1. Platform.dispatch_key holds it; the default is CPU.

Default at vllm/platforms/interface.py:142-145; CUDA sets "CUDA" (vllm/platforms/cuda.py:230).

E#

E2E latency also: end-to-end latency, E2EL, request latency

End-to-end latency. The time from the arrival of a request to its last output token.

vLLM measures it in the front end from arrival_time (vllm/v1/metrics/stats.py:594). vllm bench serve measures it at the client and calls it E2EL.

EAGLE also: eagle3, EAGLE-3, EagleSpeculator

A speculative method with a small draft head that reads the target model's hidden states. vLLM has eagle and eagle3.

use_eagle() is true for eagle, eagle3, mtp, dflash and dspark (vllm/config/speculative.py:2013-2017).

EC connector also: EC connectors, ECConnectorBase, encoder cache connector

An encoder-cache connector. It moves the outputs of a multimodal encoder between vLLM instances, with the same scheduler and worker roles as a KV connector.

Registry ECConnectorFactory (vllm/distributed/ec_transfer/ec_connector/factory.py:20-74); set with --ec-transfer-config.

Read more: KV connectors

embedding endpoint also: /v1/embeddings, embeddings API

POST /v1/embeddings of the OpenAI-compatible server. It exists only when the model supports the embed task, or when its selected task is token_embed.

Routes are registered in register_pooling_api_routers (vllm/entrypoints/pooling/factories.py:110-143).

encoder cache also: EncoderCacheManager, EncoderCache, encoder cache size

Storage for multimodal encoder outputs, keyed by the item hash. The scheduler budgets it in embeddings; the worker keeps the tensors.

Scheduler side EncoderCacheManager (vllm/v1/core/encoder_cache_manager.py:19); worker side EncoderCache in Model Runner V2 (vllm/v1/worker/gpu/mm/encoder_cache.py:8-13).

endpoint plugin also: endpoint plugins, vllm.endpoint_plugins

A plugin in the group vllm.endpoint_plugins that adds HTTP routes to the API server. It loads only when VLLM_PLUGINS names it.

The entry point gives a factory that returns an object with name, required_tasks, attach_router() and init_state() (vllm/plugins/endpoint_plugins/interface.py:49-94).

Read more: Python entry points

entry-point group also: entry point group, group

The name that collects related entry points in package metadata, for example console_scripts. Code asks importlib.metadata for all entry points of one group.

vLLM reads its own groups, for example vllm.general_plugins. Those are the plugin groups.

Read more: Python entry points

entry-point metadata also: entry_points.txt, dist-info metadata

The file entry_points.txt in the .dist-info folder of an installed package. importlib.metadata reads it when vLLM asks for the entry points of a group.

The installer writes it from entry_points= in setup.py or from the [project.entry-points."..."] tables in pyproject.toml (pyproject.toml:45-47).

Read more: Python entry points

environment variable also: environment variables, env var, env vars

A named text value that a process gets from its parent when it starts. vLLM declares its VLLM_* variables in vllm/envs.py.

Each variable is a function in the dict environment_variables (vllm/envs.py:618).

F#

forward context also: ForwardContext, set_forward_context

An object that vLLM sets before each forward pass. Attention layers read their metadata for the step from it, by layer name.

forward_native also: native implementation, forward_native()

The PyTorch-only method of a CustomOp. vLLM calls it when the op is disabled, and the other forward_* methods fall back to it by default.

Base method at vllm/model_executor/custom_op.py:136-142; disabled path at vllm/model_executor/custom_op.py:189-192.

free block queue also: FreeKVCacheBlockQueue, free_block_queue

A doubly linked list of all free KV blocks in eviction order. The pool takes new blocks from the front. Freed blocks without a hash go to the front, cached blocks to the end.

FreeKVCacheBlockQueue (vllm/v1/core/kv_cache_utils.py:258-279); used in BlockPool.free_blocks (vllm/v1/core/block_pool.py:776-807).

front end also: front-end process, frontend

The API server process of vllm serve. It runs the HTTP server and AsyncLLM, and sends requests to the engine core process.

full CUDA graph also: FULL, full cudagraph, FULL_DECODE_ONLY

One CUDA graph for the whole forward pass, attention included. Model Runner V2 replays it with a single call for batches that match a captured shape.

Captured with torch.cuda.graph in CudaGraphManager.capture and replayed in run_fullgraph (vllm/v1/worker/gpu/cudagraph_utils.py:555-568).

FusedMoE also: FusedMoEFactory, MoERunner, MoE layer

vLLM's MoE layer. FusedMoEFactory builds a router, a RoutedExperts object with the expert weights, and an MoERunner that runs the layer.

vllm/model_executor/layers/fused_moe/layer.py:92-470.

FX graph also: torch.fx, GraphModule

A PyTorch data structure (torch.fx.GraphModule) that lists the operations of a function as nodes in order. Compiler passes read and change it.

G#

gather_output also: gather output

A flag of ColumnParallelLinear. When true and TP > 1, the layer all-gathers the output slices so every rank has the full output. Default false.

vllm/model_executor/layers/linear.py:625-627.

general plugin also: general plugins, vllm.general_plugins

A function in the entry-point group vllm.general_plugins. vLLM calls it with no arguments, at most once in each process, usually to register something.

Loaded by load_general_plugins() (vllm/plugins/__init__.py:77-90). vLLM's own LoRA resolvers are general plugins (pyproject.toml:45-47).

generation config also: generation_config.json, --generation-config

The generation_config.json file of a model folder. It gives default sampling values (for example temperature, top_p) and EOS token ids. vLLM uses it by default.

H#

hash chain also: block hash, block hashes, chained hash

A list of hashes where each hash also includes the hash before it. In prefix caching, the hash of a block names the whole prefix up to the end of that block.

vLLM hashes the tuple (parent hash, block token ids, extra keys) for each full block (vllm/v1/core/kv_cache_utils.py:707-737).

I#

idx_mapping also: idx mapping, batch_idx -> req_state_idx

An int64 GPU tensor [num_reqs] that maps each row of this step's batch to the request slot in the persistent state. GPU kernels use it to gather per-request data.

Built from the sorted request ids in gather_batch_req_state and copied to the GPU in prepare_inputs (vllm/v1/worker/gpu/model_runner.py:1392-1395).

incremental detokenization also: detokenizer, DecodeStream

Turning token ids into text one token at a time, so each new token gives the next piece of text. vLLM uses the DecodeStream of the tokenizers library.

vllm/v1/engine/detokenizer.py:179-198

Inductor also: TorchInductor, torch._inductor

The default code generator of torch.compile. It turns an FX graph into fused kernels, for example Triton kernels on NVIDIA GPUs.

input thread also: process_input_sockets, input_queue

A daemon thread in the engine core process. It receives requests on the ZMQ sockets, decodes them, builds the Request object and puts it into input_queue.

EngineCoreProc.process_input_sockets (vllm/v1/engine/core.py:1921-2028).

InputProcessor also: input processor

The front-end object that checks a rendered prompt and its sampling parameters and makes an EngineCoreRequest.

IntermediateTensors also: intermediate tensors

A dict of named tensors that a pipeline stage returns instead of hidden states. For Llama: hidden_states and residual, each [num_tokens, hidden_size].

vllm/sequence.py:12-28; Llama returns it at vllm/model_executor/models/llama.py:443-450.

IO processor plugin also: IO processor, IOProcessor, vllm.io_processor_plugins

A plugin in the group vllm.io_processor_plugins. It changes the input and the output of a pooling model, for example an image in and an image out.

The function behind the entry point returns the qualified name of an IOProcessor subclass, or None (vllm/plugins/io_processors/__init__.py:63-69). The image example is from the design docs (docs/design/io_processor_plugins.md:3).

J#

json_schema response format also: response_format, json_object, JSON mode

The OpenAI field response_format of type json_schema. vLLM turns it into a structured outputs json constraint, so the answer must match the schema.

vllm/entrypoints/generate/base/protocol.py:181-211

K#

kernel provider also: provider, providers, IR provider

A named implementation of an IR op, for example vllm_c, oink, aiter or triton. The name native is reserved for the PyTorch version, and the name unfused is also reserved.

register_impl at vllm/ir/op.py:240-298; reserved names at vllm/ir/op.py:36.

KV cache group also: KVCacheGroupSpec, kv cache groups

A set of layers that use the same KV cache layout and share one block table, for example all full-attention layers.

KV cache quantization also: FP8 KV cache, kv-cache-dtype, fp8 KV

Storing the keys and values of the KV cache in a smaller format, for example FP8, with scales. FP8 halves the KV memory per token compared with BF16. Set with --kv-cache-dtype.

Read more: KV connectors

KV consumer also: kv_consumer, decode instance, decoder

An instance with kv_role kv_consumer: in disaggregated prefill it loads the prompt's KV blocks from a producer and then decodes.

vllm/config/kv_transfer.py:11-12, vllm/config/kv_transfer.py:134-136.

KV events also: KV cache events, --kv-events-config, BlockStored, BlockRemoved, KVEventsConfig

Messages that an engine core publishes when its prefix cache changes: BlockStored, BlockRemoved and AllBlocksCleared. An external router can subscribe over ZMQ to know which prefixes each engine holds.

Event types (vllm/distributed/kv_events.py:56-142); KVEventsConfig (vllm/config/kv_events.py:10-52); publish per step (vllm/v1/core/sched/scheduler.py:2469-2484).

KV offloading also: KV cache offloading, OffloadingConnector, --kv-offloading-size

Copying KV cache blocks to CPU memory or storage, so that a later request can load them again instead of computing the prefill again.

--kv-offloading-size turns it on through a KV connector (vllm/config/vllm.py:1133-1167); the native code is in vllm/v1/kv_offload/.

Read more: KV connectors

KV producer also: kv_producer, prefill instance, prefiller

An instance with kv_role kv_producer: in disaggregated prefill it computes the prompt and keeps the KV blocks until a decode instance has read them.

vllm/config/kv_transfer.py:11-12, vllm/config/kv_transfer.py:130-132.

KVCacheBlock also: KV cache block metadata, ref_cnt

The scheduler-side record of one KV cache block: block_id, ref_cnt, the block hash (only when full and cached) and two links for the free queue. The KV data itself is on the GPU.

vllm/v1/core/kv_cache_utils.py:161-186

KVTransferConfig also: --kv-transfer-config, kv_role, kv_connector_extra_config

The config that turns on a KV connector: kv_connector (the name), kv_role (kv_producer, kv_consumer or kv_both) and kv_connector_extra_config. Set with --kv-transfer-config.

Defined at vllm/config/kv_transfer.py:56-91; a connector name without a role raises (vllm/config/kv_transfer.py:120-124).

Read more: KV connectors

L#

lazy registration also: lazy entry, _LazyRegisteredModel, lazy model

A registry entry that stores only a module name and a class name. vLLM imports the module only when a process needs the class.

register_model with a "module:Class" string makes a _LazyRegisteredModel (vllm/model_executor/models/registry.py:1187-1193). All built-in entries are lazy.

load format also: --load-format, load_format

The value of --load-format (default auto). It selects the model loader class and, for the default loader, which weight files to read.

Table of the 15 built-in formats: vllm/model_executor/model_loader/__init__.py:50-66.

load-balancing mode also: internal LB, external LB, hybrid LB, DP load balancing

How requests reach the DP ranks: internal (the API servers pick an engine), hybrid (per node), or external (one endpoint per rank, an outside router picks).

Mode detection at vllm/entrypoints/cli/serve.py:79-103.

logits processor also: logits processors, LogitsProcessor, --logits-processors, vllm.logits_processors

Code that changes the logits of the batch before sampling. Model Runner V2 runs built-in ones (bias, penalties, bad words) and custom ones.

V2 interface: add_request(slot), apply_staged_writes(), apply(logits, ctx) (vllm/v1/worker/gpu/sample/logits_processor/interface.py:92-158).

logprobs also: prompt_logprobs, top_logprobs, logprobs_mode

Log probabilities of the sampled token and the top alternatives, returned per output token (logprobs) or per prompt token (prompt_logprobs).

Capped by --max-logprobs, default 20 (vllm/config/model.py:286); raw or processed values by --logprobs-mode (vllm/config/model.py:291-299).

long_prefill_token_threshold also: --long-prefill-token-threshold

A cap on the tokens that one request can get in one step. 0 (the default) means no cap. The scheduler ignores it when only one request is running or waiting.

Field: vllm/config/scheduler.py:79-84; applied at vllm/v1/core/sched/scheduler.py:677-683 and vllm/v1/core/sched/scheduler.py:764-765.

LoRA adapter also: LoRA, adapter

A small set of extra low-rank weights that changes a base model for one task. One server can apply different adapters to different requests.

M#

Mamba state also: MambaSpec, MambaManager, mamba_cache_mode, SSM state

The fixed-size recurrent state of a Mamba (state-space) layer. It replaces a growing KV cache. vLLM stores it in KV blocks; prefix caching keeps states at block boundaries only in mamba_cache_mode align.

MambaSpec (vllm/v1/kv_cache_interface.py:1079-1151); MambaManager (vllm/v1/core/single_type_kv_cache_manager.py:1424-1464); mamba_cache_mode (vllm/config/cache.py:301-308).

max_concurrent_batches also: batch queue size

The number of engine steps that can be in flight at the same time. It sets the size of the batch queue: pp_size + 1 with async scheduling and Model Runner V2, so 2 on one GPU.

VllmConfig.max_concurrent_batches (vllm/config/vllm.py:586-597).

maximum concurrency also: Maximum concurrency, kv_cache_max_concurrency

The start-up log value: KV blocks divided by the blocks that one request of max_model_len needs. It tells how many full-length requests fit in the KV cache at once.

Computed in get_max_concurrency_for_kv_cache_config (vllm/v1/core/kv_cache_utils.py:1015-1047) and logged as "Maximum concurrency for N tokens per request: X.XXx" (vllm/v1/core/kv_cache_utils.py:1090-1108).

MessageQueue also: shared-memory message queue, rpc_broadcast_mq, shm_broadcast

vLLM's one-writer, many-reader queue in shared memory (a ring buffer of chunks). Large messages go over a ZMQ socket instead. The multiproc executor uses it to talk to its workers.

vllm/distributed/device_communicators/shm_broadcast.py:483-530; large messages: vllm/distributed/device_communicators/shm_broadcast.py:942-946.

metadata builder also: AttentionMetadataBuilder, builder

The AttentionMetadataBuilder of a backend. It turns the common batch description into the backend's own attention metadata, once per step for each group of layers.

Abstract class at vllm/v1/attention/backend.py:592-761; the backend names it in get_builder_cls().

middleware also: --middleware, ASGI middleware

Code that runs around every HTTP request before the route, for example CORS or the API-key check. vLLM adds it in init_entrypoints_middleware.

vllm/entrypoints/serve/middleware/register.py:19-75

model implementation also: model_impl, --model-impl, Transformers backend fallback

The --model-impl setting: auto (vLLM class, else the Transformers backend), vllm, transformers or terratorch.

Values at vllm/config/model.py:109; used in the lookup order of vllm/model_executor/models/registry.py:1351-1402.

model info also: _ModelInfo, modelinfo, modelinfos

_ModelInfo: a frozen record of 24 capability values of a model class (generative, pooling, PP, LoRA, multimodal, hybrid ...). The registry computes it from the class.

Fields at vllm/model_executor/models/registry.py:891-915; computed by from_model_cls (vllm/model_executor/models/registry.py:917-954); cached as JSON in VLLM_CACHE_ROOT/modelinfos.

model loader also: model loaders, BaseModelLoader, --load-format, register_model_loader

A class that builds the model and loads its weights for one --load-format, for example safetensors, dummy or tensorizer.

Base class BaseModelLoader; table _LOAD_FORMAT_TO_MODEL_LOADER and register_model_loader (vllm/model_executor/model_loader/__init__.py:50-128).

model registry also: ModelRegistry, model registry, _ModelRegistry

ModelRegistry: the one table per process that maps an architecture name from config.json, such as LlamaForCausalLM, to a model class.

Built from the tables in vllm/model_executor/models/registry.py:1565-1573; plugins add entries with register_model (vllm/model_executor/models/registry.py:1160-1203).

model runner also: Model Runner V2, MRV2

The object in a worker that prepares the inputs of a step, runs the model and samples tokens. vLLM uses Model Runner V2 by default.

modular MoE kernel also: FusedMoEKernel, prepare/finalize

An MoE kernel built from two parts: a prepare/finalize object (quantize and dispatch, then combine) and an experts object (the grouped GEMMs).

Design comment at vllm/model_executor/layers/fused_moe/modular_kernel.py:51-84.

MPClient also: multiprocess client, AsyncMPClient, SyncMPClient

The base class of the engine core clients that talk to engine core processes over ZMQ. It binds (or takes over) the sockets, waits for the ready messages and, when it started the engine processes, watches them.

msgpack also: MessagePack

MessagePack, a compact binary format for data. vLLM encodes the messages between the front end and the engine core with it.

MTP also: multi-token prediction, deepseek_mtp

Multi-token prediction: extra layers in the checkpoint that predict the next tokens. vLLM uses them as a drafter with the method mtp.

Model-specific names such as deepseek_mtp are deprecated and become mtp (vllm/config/speculative.py:1197-1201).

multi-LoRA serving also: multi-LoRA, multi LoRA, batched LoRA

One server keeps one base model and several LoRA adapters. It runs requests with different adapters in the same batch; each token uses the adapter of its request.

N#

NONE_HASH also: NONE_HASH, init_none_hash

The value that stands for the parent hash of block 0. vLLM derives it from a seed: PYTHONHASHSEED if set, else a fixed seed for the sha256 algorithms and a random seed for the xxhash algorithms.

vllm/v1/core/kv_cache_utils.py:104-158

null block also: null_block, block 0

Block 0 of the block pool. It is never given to a request. It fills block table places that hold no data, for example blocks outside a sliding window.

vllm/v1/core/block_pool.py:184-188; KV cache usage leaves it out (vllm/v1/core/block_pool.py:888-892).

num_computed_tokens also: computed tokens

The number of tokens of a request whose KV values are in the KV cache (or are computed by a step already started). The scheduler gives tokens until it catches up with num_tokens.

Set to 0 by Request.__init__ (vllm/v1/request.py:184); advanced in _update_after_schedule (vllm/v1/core/sched/scheduler.py:1716); reset to 0 on preemption (vllm/v1/core/sched/scheduler.py:1680).

num_speculative_tokens also: k, --spec-tokens, number of speculative tokens

k, the number of draft tokens per request per step. A decode then has 1 + k query tokens.

No default; it comes from n_predict of the draft config when present (vllm/config/speculative.py:393-395).

O#

op registry also: op_registry, op_registry_oot

op_registry: the dict from a registered op name, such as rms_norm, to its CustomOp or PluggableLayer class. The +name and -name switches use these names.

vllm/model_executor/custom_op.py:16-22; filled by register (vllm/model_executor/custom_op.py:310-323).

OpenAI-compatible server also: API server, OpenAI server

The HTTP server of vllm serve. Its routes, such as /v1/chat/completions, accept the same requests as the OpenAI API.

optimization level also: -O, --optimization-level, OptimizationLevel, O2

The -O0 to -O3 switch of vllm serve. It sets defaults for the CUDA graph mode and the fusion passes; O2 is the default.

Enum at vllm/config/vllm.py:131-143, default at vllm/config/vllm.py:434, defaults per level at vllm/config/vllm.py:254-338.

out-of-tree also: OOT, out of tree

Code that is not in the vLLM repository, for example a model or a platform in your own package. Short form: OOT.

P#

P/D disaggregation also: disaggregated prefill, prefill/decode disaggregation, disagg

Running prefill and decode of a request on different vLLM instances, with a KV connector that moves the KV cache between them.

Why: docs/features/disagg_prefill.md:8-16.

page size unification also: unify_kv_cache_spec_page_size, page_size_padded, page size

Making the bytes of one block equal in every layer of a model with several KV cache groups. vLLM scales up the block size of small pages, or pads pages, because all groups share one block pool.

unify_kv_cache_spec_page_size (vllm/v1/core/kv_cache_planning.py:346-408); for Mamba hybrids also _align_hybrid_block_size (vllm/platforms/interface.py:939-1134).

percentile also: p50, p90, p99, median

The value that a given part of the samples is below. p50 (the median): half of the samples are below it. p99: 99 % are below it.

persistent batch also: persistent state, req_states, RequestState (MRV2)

The request state that Model Runner V2 keeps on the GPU between steps, one fixed row per running request. Each step gathers its inputs from these rows.

RequestState allocates max_num_reqs rows for token ids, lengths, computed counts, last sampled tokens and drafts (vllm/v1/worker/gpu/states.py:9-85). The design doc explains why the state is kept apart from the per-step inputs (docs/design/model_runner_v2.md:31-39).

piecewise CUDA graph also: PIECEWISE, piecewise cudagraph, piecewise graph

CUDA graphs captured for the compiled pieces between attention ops. Attention itself runs eagerly between the replays, so any batch shape works.

Each compiled piece gets a CUDAGraphWrapper with runtime mode PIECEWISE (vllm/compilation/backends.py:656-707).

pipeline bubble also: bubble, bubbles, pipeline bubbles

Idle time of a pipeline stage. It waits for the stage before it, because there is not enough work in flight to keep every stage busy.

pipeline parallelism also: PP, pipeline parallel

Each GPU holds a different range of layers. The hidden states go from one GPU to the next (PP).

placeholder token also: placeholders, image token, mm_placeholders

A prompt token that stands for one embedding of an image, audio or video item. The model replaces it with encoder output.

The processor inserts as many placeholders as the encoder gives embeddings (vllm/multimodal/processing/processor.py:1816-1826).

platform also: current_platform, Platform class

The object that holds the hardware choices of a process (CUDA, ROCm, CPU, ...). Code reads it as current_platform.

platform detector also: platform detectors, detector function, builtin_platform_plugins

A function with no arguments that returns the class name of a Platform subclass when its hardware is present, else None. vLLM has five built-in detectors.

Built-in detectors: tpu, cuda, rocm, xpu, cpu (vllm/platforms/__init__.py:231-237). An out-of-tree platform plugin is also a detector (vllm/platforms/__init__.py:254-259).

platform hook also: platform hooks, config hook

A method of the Platform class that vLLM calls at a fixed point of the start-up or of a request, for example check_and_update_config or get_attn_backend_cls.

Q#

QKVParallelLinear also: qkv_proj, QKV parallel linear

The column-parallel layer that computes Q, K and V in one GEMM. It splits by attention heads and copies KV heads when TP is larger than the KV head count.

vllm/model_executor/layers/linear.py:1011-1091.

quantization method also: QuantizationConfig, quantization

A vLLM class pair that reads a quantized checkpoint and runs its layers, for example FP8, AWQ or GPTQ. --quantization selects it.

QuantizationConfig also: quantization config, quant_config

The base class of a quantization method. One object holds the checkpoint settings and returns a quantize method for each layer through get_quant_method.

Abstract methods: get_name, get_supported_act_dtypes, get_min_capability, get_config_filenames, from_config, get_quant_method (vllm/model_executor/layers/quantization/base_config.py:109-217).

quantize method also: QuantizeMethodBase, quant_method, LinearMethodBase

The per-layer object that makes a layer's quantized weights (create_weights), runs its forward (apply) and post-processes the loaded weights.

Base class QuantizeMethodBase (vllm/model_executor/layers/quantization/base_config.py:23-95). A linear layer stores it as self.quant_method (vllm/model_executor/layers/linear.py:290-296).

R#

rank layout also: rank grid, layout order

The order in which vLLM arranges all ranks before it cuts them into groups: ExternalDP × DP × PP × PCP × TP, with TP as the fastest dimension.

vllm/distributed/parallel_state.py:2148-2163.

readiness probe also: liveness probe, health probe

A periodic health request from an orchestrator (for example Kubernetes) that decides when a server gets traffic. For vllm serve it is usually GET /health.

/health answers 200, or 503 after EngineDeadError (vllm/entrypoints/serve/instrumentator/health.py:22-33). The vLLM docs use /health for both probes (docs/deployment/k8s.md:244-254).

reasoning output also: reasoning_content, reasoning field, reasoning content

The thinking text of a reasoning model. With --reasoning-parser, vLLM returns it in the vLLM field reasoning (old name reasoning_content), not in content.

Response field at vllm/entrypoints/openai/chat_completion/protocol.py:79-80; old input name renamed at vllm/entrypoints/openai/chat_completion/protocol.py:590-592.

reasoning parser also: reasoning parsers, ReasoningParserManager, --reasoning-parser

A class that splits the model's reasoning text from its answer. --reasoning-parser selects it; the API server and the engine core both use it.

Registry ReasoningParserManager (vllm/reasoning/abs_reasoning_parsers.py:225-373).

recompute preemption also: recompute, PREEMPTED

vLLM's only kind of preemption: the scheduler frees all KV blocks of the request and sets num_computed_tokens to 0. Later the request computes its tokens again (prefix caching can give some back).

Scheduler._preempt_request (vllm/v1/core/sched/scheduler.py:1658-1701).

reduce_results also: reduce results

A flag of RowParallelLinear (default true). When true and TP > 1, the layer all-reduces the partial sums of the ranks.

vllm/model_executor/layers/linear.py:1803-1806.

register_oot also: register_oot(), @CustomOp.register_oot, out-of-tree op

The decorator that registers an out-of-tree replacement class for an in-tree CustomOp or PluggableLayer. vLLM then builds the replacement wherever code builds the in-tree class.

vllm/model_executor/custom_op.py:334-356. A string as the first argument raises TypeError; use name="...".

rejection sampler also: RejectionSampler, verification

The sampler that checks draft tokens against the target model's distribution, keeps the accepted prefix and adds one recovered or bonus token.

Model Runner V2 class at vllm/v1/worker/gpu/spec_decode/rejection_sampler.py:147; variants standard, synthetic, block, fly (vllm/config/speculative.py:83).

ReplicatedLinear also: replicated linear

A linear layer that every TP rank holds in full. It splits nothing and needs no communication.

vllm/model_executor/layers/linear.py:327-431.

request also: requests

One generation job from a client: a prompt, sampling settings and the output tokens so far. vLLM gives each request an id and tracks it until it finishes.

S#

safetensors also: .safetensors, model.safetensors.index.json, safetensors index

A file format for tensors: a JSON header with the name, dtype, shape and byte offsets of each tensor, then the raw bytes. Loading it runs no code.

T#

tensor parallelism also: TP, tensor parallel

Each GPU holds a slice of every weight matrix. The GPUs compute their slices of each layer and then combine the results (TP).

token budget also: max_num_batched_tokens, max_num_scheduled_tokens

The maximum number of tokens that the scheduler puts into one step. It is max_num_scheduled_tokens, which is max_num_batched_tokens by default. Prefill and decode tokens share it.

token throughput also: tokens per second, tok/s, output throughput, generation throughput

The number of tokens per second that a server makes (output tokens), or reads and makes (input plus output tokens).

token-in token-out also: TITO, tokens in tokens out, /inference/v1/generate

Serving where the client sends token ids and gets token ids back, with no template and no detokenization in the server. vLLM has /inference/v1/generate for it.

Registered only with --enable-scale-out or --tokens-only (vllm/entrypoints/scale_out/factories.py:94-137).

tool parser also: tool parsers, ToolParserManager, --tool-call-parser

A class in the API server that turns the model's text into OpenAI tool calls. --tool-call-parser selects it by name.

Registry ToolParserManager (vllm/tool_parsers/abstract_tool_parser.py:235-380).

tool_choice also: tool choice, tool_choice auto, tool_choice required

The request field that says whether the model must call a tool: none, auto, required, or one named function. vLLM enforces required and named with structured outputs.

Validation at vllm/entrypoints/openai/chat_completion/protocol.py:959-1043; schema for required or named at vllm/tool_parsers/utils.py:449-490.

torch.compile also: Dynamo, Inductor

The PyTorch compiler. It captures a model as a graph and makes faster code for it. vLLM adds its own backend and passes.

torch.library op also: custom operator, torch.ops, registered op

An operator that code registers with PyTorch under a namespace, for example torch.ops.vllm.unified_attention_with_output. torch.compile treats it as one opaque node.

touch also: touch(), BlockPool.touch

BlockPool.touch(): add 1 to the ref_cnt of a prefix-cache hit block. A block with ref_cnt 0 first leaves the free block queue, so it cannot be evicted.

vllm/v1/core/block_pool.py:752-770

TPOT also: time per output token

Time per output token. The mean time between output tokens of one request, after the first token.

TTFT also: time to first token

Time to first token. The time from the arrival of a request to the first output token. Prefill and queue time set it.

U#

uni executor also: UniProcExecutor, uniproc executor

The executor backend "uni" (UniProcExecutor). It makes the one worker as an object in the engine core process and calls its methods directly. By default, vLLM uses it only when TP = PP = DP = 1.

vllm/v1/executor/uniproc_executor.py:51-75; chosen at vllm/config/parallel.py:1055 and vllm/config/parallel.py:1099-1100.

UnspecifiedPlatform also: unspecified platform, PlatformEnum.UNSPECIFIED

The fallback platform when no detector finds hardware. It has an empty device_type, so most device calls fail or return nothing.

Defined at vllm/platforms/interface.py:1493-1495; chosen at vllm/platforms/__init__.py:290-292.

usage context also: UsageContext, OPENAI_API_SERVER, LLM_CLASS

A UsageContext enum value that tells create_engine_config() who starts the engine. vllm serve passes OPENAI_API_SERVER; the token budget defaults depend on it.

Enum at vllm/usage/usage_lib.py:111-117; used for defaults at vllm/engine/arg_utils.py:3159-3168.

V#

V1 engine also: V1, vllm/v1

The only engine of vLLM at this commit: the code in vllm/v1/ (AsyncLLM, the engine core, the scheduler, the workers). The old V0 names are aliases of it.

W#

W8A8 also: W8A8-FP8, W8A8-INT8, weight and activation quantization

Weights and activations both in 8 bits (FP8 or INT8), so the matrix multiply itself runs in 8 bits. Each tensor has scales that map it back to real values.

Z#

ZMQ also: ZeroMQ, pyzmq

ZeroMQ, a message library. vLLM uses ZMQ sockets to send requests and outputs between the front end and the engine core processes.

ZMQ socket type also: ROUTER, DEALER, PUSH, PULL, XPUB

The pattern of a ZMQ socket, for example ROUTER, DEALER, PUSH, PULL or XPUB. The type sets who can send, who can receive and how messages are routed.