Config classes

Referenceabout 89 min read

Code version: vLLM @ 3e44c0d · 2026-10-22

In one sentence

Each part of the engine configuration is one @config dataclass in vllm/config/, and one VllmConfig object holds the top-level ones.

Use this page in two ways. To learn the mechanism, read the first four sections. To look up one class or field, go to All config classes, or search for its name. The script tools/gen_config_classes.py makes the tables from the source of vllm/config/*.py. The text above the tables is written by hand from the code.

The @config decorator#

All config classes use the decorator config from vllm/config/utils.py. It makes a config class: a pydantic dataclass whose settings forbid extra fields:

vllm/config/utils.py:70-82GitHub ↗

    # Extra fields are forbidden by default
    merged_config = ConfigDict(extra="forbid")
    if config is not None:
        merged_config.update(config)

    def decorator(cls: type[ConfigT]) -> type[ConfigT]:
        return dataclass(cls, config=merged_config, **kwargs)  # type: ignore[return-value]

    # Called with arguments: @config(config=...)
    if cls is None:
        return decorator
    # Called without arguments: @config
    return decorator(cls)

dataclass here is pydantic.dataclasses.dataclass (vllm/config/utils.py:20). This has three effects:

Unknown keys are an error. extra="forbid" rejects a key that is not a field. A JSON flag such as --compilation-config is parsed with pydantic, and a ValidationError becomes an argparse error (vllm/engine/arg_utils.py:369-374). So a misspelled key stops vllm serve at start-up.
Field limits are checked. Fields can use Field(default=..., ge=1), and pydantic checks the value. For example, gpu_memory_utilization must be above 0 and at most 1 (vllm/config/cache.py:212).
A class can pass its own pydantic settings. VllmConfig uses @config(config=ConfigDict(arbitrary_types_allowed=True)) (vllm/config/vllm.py:347).

The docstring of the decorator says that all config classes in vLLM should use it (vllm/config/utils.py:61).

Field docstrings are the help text#

A config class documents each field with a string literal right after it. Python ignores these strings, but get_attr_docs reads them from the source with ast (vllm/config/utils.py:161-198, vllm/config/utils.py:201-215). The vllm serve --help text comes from them (vllm/engine/arg_utils.py:355-356). This is SchedulerConfig:

vllm/config/scheduler.py:24-53GitHub ↗

@config
class SchedulerConfig:
    """Scheduler configuration."""

    max_model_len: InitVar[int]
    """Maximum length of a sequence (including prompt and generated text).

    Note: This is stored in the ModelConfig, and is used only here to
    provide fallbacks and validate other attributes."""

    is_encoder_decoder: InitVar[bool]
    """True if the model is an encoder-decoder model.

    Note: This is stored in the ModelConfig, and is used only here to
    disable chunked prefill and prefix caching for encoder-decoder models.
    """

    DEFAULT_MAX_NUM_BATCHED_TOKENS: ClassVar[int] = 2048
    DEFAULT_MAX_NUM_BATCHED_TOKENS_FOR_BATCHED_DP: ClassVar[int] = 256
    DEFAULT_MAX_NUM_SEQS: ClassVar[int] = 128

    runner_type: RunnerType = "generate"
    """The runner type to launch for the model."""

    max_num_batched_tokens: int = Field(default=DEFAULT_MAX_NUM_BATCHED_TOKENS, ge=1)
    """Maximum number of tokens that can be processed in a single iteration.

    The default value here is mainly for convenience when testing.
    In real usage, this should be set in `EngineArgs.create_engine_config`.
    """

InitVar fields are arguments of __init__ only. ClassVar lines are class constants, not fields. The docstring of max_num_batched_tokens says that the real value comes from EngineArgs.create_engine_config.

The generated tables mark these two kinds. An InitVar is not in dataclasses.fields(), so get_kwargs of its class makes no flag for it (vllm/engine/arg_utils.py:329). For example, ModelConfig takes the multimodal settings as InitVar arguments and builds a MultiModalConfig from them (vllm/config/model.py:432-434, vllm/config/model.py:572-577, vllm/config/model.py:881). The flags, for example --limit-mm-per-prompt, come from the fields of MultiModalConfig (vllm/engine/arg_utils.py:1489-1491).

VllmConfig: the container#

VllmConfig has one attribute for each sub-config, plus some top-level settings (vllm/config/vllm.py:347-457):

vllm/config/vllm.py:347-368GitHub ↗

@config(config=ConfigDict(arbitrary_types_allowed=True))
class VllmConfig:
    """Dataclass which contains all vllm-related configuration. This
    simplifies passing around the distinct configurations in the codebase.
    """

    # TODO: use default_factory once default constructing ModelConfig doesn't
    # try to download a model
    model_config: ModelConfig = None  # type: ignore[assignment]
    """Model configuration."""
    cache_config: CacheConfig = Field(default_factory=CacheConfig)
    """Cache configuration."""
    parallel_config: ParallelConfig = Field(default_factory=ParallelConfig)
    """Parallel configuration."""
    scheduler_config: SchedulerConfig = Field(
        default_factory=SchedulerConfig.default_factory,
    )
    """Scheduler configuration."""
    device_config: DeviceConfig = Field(default_factory=DeviceConfig)
    """Device configuration."""
    load_config: LoadConfig = Field(default_factory=LoadConfig)
    """Load configuration."""

Most attributes have a default factory, so VllmConfig() makes default sub-configs. model_config is None by default, because a default ModelConfig would try to download a model (vllm/config/vllm.py:353-355). Optional features such as lora_config and speculative_config are None when they are off (vllm/config/vllm.py:381-384). For vllm serve, EngineArgs.create_engine_config builds each sub-config and then calls VllmConfig(...) (vllm/engine/arg_utils.py:2870). See vllm serve flags for the flag side.

What __post_init__ does#

VllmConfig.__post_init__ runs after pydantic has made the object. It is about 830 lines long (vllm/config/vllm.py:1492-2318). Its docstring says: "Verify configs are valid & consistent with each other." (vllm/config/vllm.py:1493). These are the main steps, in code order:

It sets instance_id from the clock (vllm/config/vllm.py:1495).
It turns off prefix caching for an instance that runs only the multimodal encoder (vllm/config/vllm.py:1505-1516).
It runs the per-architecture config hooks of the model in try_verify_and_update_config (vllm/config/vllm.py:1521, vllm/config/vllm.py:2714-2733).
It decides async scheduling when the flag is not set (vllm/config/vllm.py:1664-1698).
With enforce_eager, it turns off torch.compile and CUDA graphs (vllm/config/vllm.py:1745-1747).
It applies the platform defaults, picks the compilation mode from the optimization level, and applies the defaults of that level (vllm/config/vllm.py:1804-1810, vllm/config/vllm.py:1833-1834).
It calls the platform hook current_platform.check_and_update_config(self), which can change any part of the config (vllm/config/vllm.py:2046).
It resolves the KV transfer settings, for example the connector name for KV offloading (vllm/config/vllm.py:2197-2199).

compute_hash: one key for compiled code#

VllmConfig.compute_hash() collects the compute_hash() of each sub-config and adds the vLLM version. It returns the first 10 hex digits of a hash of this list (vllm/config/vllm.py:474-483, vllm/config/vllm.py:572-575). Its docstring says that the hash covers the configs that change the computation graph (vllm/config/vllm.py:460-468). Each sub-config decides which fields count. For example, SchedulerConfig adds max_num_batched_tokens, because LoRA buffer sizes in the compiled graph and the index width that Inductor selects depend on it (vllm/config/scheduler.py:271-280).

The torch.compile backend uses the hash. When compilation_config.cache_dir is not set (vllm/compilation/backends.py:1086), it hashes four values together: the environment hash, the config hash, the code hash and the compiler hash. The result is the name of the cache folder under VLLM_CACHE_ROOT/torch_compile_cache/ (vllm/compilation/backends.py:1059-1063, vllm/compilation/backends.py:1091-1097). So when you change --max-num-batched-tokens, the next start compiles again into a new folder.

How to read the tables#

All config classes lists each class with its file and line, the number of fields, and the first paragraph of the class docstring.
What each VllmConfig attribute holds maps each attribute to its class. quant_config holds a QuantizationConfig, which is not a @config class.
One table per class lists each annotated line of the class body: name, type and default as source text. The last column quotes the first paragraph of the field docstring as it is. (InitVar) and (ClassVar) mark lines that are not dataclass fields.
Defaults are dataclass defaults. They are not always the values that vllm serve uses. EngineArgs and __post_init__ change many of them. The table Flags you tune first gives the real defaults of the tuning fields.
Inherited fields are not copied into a subclass table. Only vllm/config/*.py is scanned, so FrontendArgs (in vllm/entrypoints/common/cli_args.py) is not here; its flags are in vllm serve flags.

All config classes#

42 classes with the @config decorator in 34 files of vllm/config/, with 543 fields. The decorator makes a pydantic dataclass that forbids extra fields (config/utils.py:52).

All config classes#

Class	Defined in	Fields	Docstring (first paragraph)
`HiSparseConfig`	`config/attention.py:18`	2	Configuration for HiSparse sparse-MLA KV offloading.
`AttentionConfig`	`config/attention.py:37`	24	Configuration for attention mechanisms in vLLM.
`AuxOutputConfig`	`config/aux_output.py:11`	2	Configuration for auxiliary-output delivery.
`CacheConfig`	`config/cache.py:179`	37	Configuration for the KV cache.
`PassConfig`	`config/compilation.py:114`	13	Configuration for custom Inductor passes.
`DynamicShapesConfig`	`config/compilation.py:343`	3	Configuration to control/debug torch compile dynamic shapes.
`CompilationConfig`	`config/compilation.py:383`	38	Configuration for compilation.
`DeviceConfig`	`config/device.py:17`	2	Configuration for the device to use for vLLM execution.
`DiffusionConfig`	`config/diffusion.py:11`	2	Configuration for discrete diffusion language models (dLLMs).
`EncoderCacheManagerConfig`	`config/ec_manager_config.py:13`	2
`ECTransferConfig`	`config/ec_transfer.py:16`	10	Configuration for distributed EC cache transfer.
`EngramConfig`	`config/engram.py:34`	4	Configuration for Engram embedding storage and sharding.
`FaultToleranceConfig`	`config/fault_tolerance.py:9`	1	Configuration for fault tolerance.
`IrOpPriorityConfig`	`config/kernel.py:21`	3	Configuration for vLLM IR op priority for dispatching/lowering during the forward pass. Each member is a list of strings, which will be installed in worker init via vllm.ir.ops.<op_name>.set_default(). A single comma-separated string is accepted as well,
`KernelConfig`	`config/kernel.py:271`	10	Configuration for kernel selection and warmup behavior.
`KVEventsConfig`	`config/kv_events.py:11`	8	Configuration for KV event publishing.
`KVTransferConfig`	`config/kv_transfer.py:57`	9	Configuration for distributed KV cache transfer.
`LoadConfig`	`config/load.py:27`	10	Configuration for loading the model weights.
`LoggingConfig`	`config/logging.py:16`	4	vLLM logging. Default ``dictConfig`` adds a formatted stream handler.
`LoRAConfig`	`config/lora.py:30`	12	Configuration for LoRA.
`MambaConfig`	`config/mamba.py:37`	4	Configuration for Mamba SSM backends.
`ModelConfig`	`config/model.py:152`	53	Configuration for the model.
`MultiModalConfig`	`config/multimodal.py:301`	24	Controls the behavior of multimodal models.
`ObservabilityConfig`	`config/observability.py:20`	14	Configuration for observability - metrics and tracing.
`UVAOffloadConfig`	`config/offload.py:16`	2	Configuration for UVA (Unified Virtual Addressing) CPU offloading.
`PrefetchOffloadConfig`	`config/offload.py:48`	4	Configuration for prefetch-based CPU offloading.
`OffloadConfig`	`config/offload.py:80`	3	Configuration for model weight offloading to reduce GPU memory usage.
`EPLBConfig`	`config/parallel.py:64`	9	Configuration for Expert Parallel Load Balancing (EP).
`ParallelConfig`	`config/parallel.py:137`	64	Configuration for the distributed execution.
`PoolerConfig`	`config/pooler.py:30`	13	Controls the behavior of output pooling in pooling models.
`ProfilerConfig`	`config/profiler.py:67`	25	Dataclass which contains profiler config for the engine.
`QuantSpec`	`config/quantization.py:78`	2	Quantization spec for one layer kind (linear or MoE).
`QuantizationConfigArgs`	`config/quantization.py:108`	4	User-facing quantization configuration.
`ReasoningConfig`	`config/reasoning.py:13`	7	Configuration for reasoning models.
`SchedulerConfig`	`config/scheduler.py:25`	22	Scheduler configuration.
`SpeculativeConfig`	`config/speculative.py:387`	41	Configuration for speculative decoding.
`SpeechToTextConfig`	`config/speech_to_text.py:55`	4	Configuration for speech-to-text models.
`StructuredOutputsConfig`	`config/structured_outputs.py:17`	6	Dataclass which contains structured outputs config for the engine.
`VllmConfig`	`config/vllm.py:348`	34	Dataclass which contains all vllm-related configuration. This simplifies passing around the distinct configurations in the codebase.
`WatchdogConfig`	`config/watchdog_config.py:13`	3	Configuration for the stack-dump watchdog.
`WatermarkConfig`	`config/watermarking.py:27`	8	Configuration for text watermark generation.
`WeightTransferConfig`	`config/weight_transfer.py:9`	1	Configuration for weight transfer during RL training.

What each `VllmConfig` attribute holds#

Attribute	Holds	Type	Default
`model_config` `config/vllm.py:355`	`ModelConfig`	`ModelConfig`	`None`
`cache_config` `config/vllm.py:357`	`CacheConfig`	`CacheConfig`	`Field(default_factory=CacheConfig)`
`parallel_config` `config/vllm.py:359`	`ParallelConfig`	`ParallelConfig`	`Field(default_factory=ParallelConfig)`
`scheduler_config` `config/vllm.py:361`	`SchedulerConfig`	`SchedulerConfig`	`Field(default_factory=SchedulerConfig.default_factory)`
`device_config` `config/vllm.py:365`	`DeviceConfig`	`DeviceConfig`	`Field(default_factory=DeviceConfig)`
`load_config` `config/vllm.py:367`	`LoadConfig`	`LoadConfig`	`Field(default_factory=LoadConfig)`
`offload_config` `config/vllm.py:369`	`OffloadConfig`	`OffloadConfig`	`Field(default_factory=OffloadConfig)`
`attention_config` `config/vllm.py:371`	`AttentionConfig`	`AttentionConfig`	`Field(default_factory=AttentionConfig)`
`aux_output_config` `config/vllm.py:373`	`AuxOutputConfig`	`AuxOutputConfig`	`Field(default_factory=AuxOutputConfig)`
`engram_config` `config/vllm.py:375`	`EngramConfig`	`EngramConfig \| None`	`None`
`mamba_config` `config/vllm.py:377`	`MambaConfig`	`MambaConfig`	`Field(default_factory=MambaConfig)`
`kernel_config` `config/vllm.py:379`	`KernelConfig`	`KernelConfig`	`Field(default_factory=KernelConfig)`
`lora_config` `config/vllm.py:381`	`LoRAConfig`	`LoRAConfig \| None`	`None`
`speculative_config` `config/vllm.py:383`	`SpeculativeConfig`	`SpeculativeConfig \| None`	`None`
`watermark_config` `config/vllm.py:385`	`WatermarkConfig`	`WatermarkConfig \| None`	`None`
`diffusion_config` `config/vllm.py:387`	`DiffusionConfig`	`DiffusionConfig \| None`	`None`
`structured_outputs_config` `config/vllm.py:390`	`StructuredOutputsConfig`	`StructuredOutputsConfig`	`Field(default_factory=StructuredOutputsConfig)`
`observability_config` `config/vllm.py:394`	`ObservabilityConfig`	`ObservabilityConfig`	`Field(default_factory=ObservabilityConfig)`
`logging_config` `config/vllm.py:398`	`LoggingConfig`	`LoggingConfig`	`Field(default_factory=LoggingConfig)`
`quant_config` `config/vllm.py:400`	`QuantizationConfig` (not a `@config` class)	`QuantizationConfig \| None`	`None`
`compilation_config` `config/vllm.py:402`	`CompilationConfig`	`CompilationConfig`	`Field(default_factory=CompilationConfig)`
`profiler_config` `config/vllm.py:411`	`ProfilerConfig`	`ProfilerConfig`	`Field(default_factory=ProfilerConfig)`
`kv_transfer_config` `config/vllm.py:413`	`KVTransferConfig`	`KVTransferConfig \| None`	`None`
`kv_events_config` `config/vllm.py:415`	`KVEventsConfig`	`KVEventsConfig \| None`	`None`
`ec_transfer_config` `config/vllm.py:417`	`ECTransferConfig`	`ECTransferConfig \| None`	`None`
`ec_manager_config` `config/vllm.py:419`	`EncoderCacheManagerConfig`	`EncoderCacheManagerConfig`	`Field(default_factory=EncoderCacheManagerConfig)`
`reasoning_config` `config/vllm.py:423`	`ReasoningConfig`	`ReasoningConfig \| None`	`None`
`additional_config` `config/vllm.py:428`	—	`dict \| SupportsHash`	`Field(default_factory=dict)`
`instance_id` `config/vllm.py:432`	—	`str`	`""`
`optimization_level` `config/vllm.py:434`	—	`OptimizationLevel`	`OptimizationLevel.O2`
`performance_mode` `config/vllm.py:440`	—	`PerformanceMode`	`"balanced"`
`weight_transfer_config` `config/vllm.py:447`	`WeightTransferConfig`	`WeightTransferConfig \| None`	`None`
`watchdog_config` `config/vllm.py:450`	`WatchdogConfig`	`WatchdogConfig`	`Field(default_factory=WatchdogConfig)`
`shutdown_timeout` `config/vllm.py:453`	—	`int`	`Field(default=0, ge=0)`

HiSparseConfig#

config/attention.py:18, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`device_buffer_size` `config/attention.py:21`	`int \| None`	`Field(default=None, gt=0)`	Total per-request GPU hot-buffer rows, including the newest-token slot.
`eager_host_mirror` `config/attention.py:29`	`bool`	`True`	Mirror decode-written KV rows to the host pool during the forward so page spills complete without moving data. When disabled, decode rows stay resident-only and evicted pages are copied to host at spill time. Prefill rows are always mirrored during the forward.

AttentionConfig#

config/attention.py:37, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`backend` `config/attention.py:40`	`AttentionBackendEnum \| None`	`None`	Attention backend to use. Use "auto" or None for automatic selection.
`minimax_m3_msa_decode_backend` `config/attention.py:43`	`MiniMaxM3MSADecodeBackend`	`"triton"`	Sparse decode kernel used by the MiniMax M3 MSA backend.
`backend_per_kind` `config/attention.py:46`	`dict[str, AttentionBackendEnum]`	`field(default_factory=dict)`	Per-KV-cache-group attention backend overrides, keyed by `KVCacheSpecKind` (e.g. `{"mla_attention": "FLASHINFER_MLA", "sliding_window_mla": "TRITON_MLA"}`). This lets a model that splits its layers across multiple KV-cache groups (e.g. interleaved full and sliding-window attention) use a different backend per group.
`flash_attn_version` `config/attention.py:57`	`Literal[2, 3, 4] \| None`	`None`	Force vllm to use a specific flash-attention version (2, 3, or 4). Only valid when using the flash-attention backend.
`flash_attn_max_num_splits_for_cuda_graph` `config/attention.py:61`	`int`	`32`	Flash Attention max number splits for cuda graph decode.
`tokenspeed_mla_min_split_kv` `config/attention.py:64`	`int`	`Field(default=1, ge=1, le=256)`	Minimum KV splits for TokenSpeed MLA decode. The default of 1 preserves automatic scheduling. Larger values request a split floor and may increase workspace memory. Only applies to the TokenSpeed MLA backend.
`tq_max_kv_splits_for_cuda_graph` `config/attention.py:69`	`int`	`32`	TurboQuant max NUM_KV_SPLITS for cuda graph decode. Fixes the split count so grid dimensions are constant across captures, and buffers can be pre-allocated to avoid inflating the memory estimate.
`use_trtllm_attention` `config/attention.py:74`	`bool \| None`	`None`	If set to True/False, use or don't use the TRTLLM attention backend in flashinfer. If None, auto-detect the attention backend in flashinfer.
`disable_flashinfer_q_quantization` `config/attention.py:78`	`bool`	`False`	If set, when using fp8 kv, do not quantize Q to fp8.
`mla_prefill_backend` `config/attention.py:81`	`MLAPrefillBackendEnum \| None`	`None`	MLA prefill backend to use. If None, will be selected automatically. Valid options: FLASH_ATTN (FA3/FA4), FLASHINFER, TRTLLM_RAGGED.
`use_prefill_query_quantization` `config/attention.py:85`	`bool`	`False`	If set, quantize query for attention in prefill.
`trtllm_ragged_backend` `config/attention.py:88`	`Literal["trtllm-gen", "cute-dsl"]`	`"trtllm-gen"`	Kernel backend for cached-context TRTLLM_RAGGED MLA prefill. CuTe DSL requires FP8 queries for head dimension 192 and does not support head dimension 256. New-token prefill always uses TRTLLM.
`indexer_kv_dtype` `config/attention.py:93`	`IndexerKVDType \| str`	`"auto"`	Data type for the sparse-attention indexer K cache. "auto" picks the model's default (bf16 for MiniMax M3, fp8 for the DeepSeek sparse indexer). Quantized formats (fp8, mxfp4, nvfp4) require indexer kernel support in the backend.
`indexer_sparse_logits` `config/attention.py:99`	`bool`	`False`	DeepSeek V4.1 two-level indexer: score only the candidate blocks with DeepGEMM's sparse MQA-logits kernels instead of computing dense logits over the whole context and masking them. Requires `indexer_kv_dtype="mxfp4"`, an SM100-class GPU, DeepGEMM >= 2.8 and the DeepSelect top-k extension (the top-k runs on the kernels' bf16 logits). The sparse path costs O(candidate blocks) per query regardless …
`indexer_tp_shard_min_rows` `config/attention.py:111`	`int`	`Field(default=65536, ge=1)`	Minimum total prefill rows for the sparse-attention indexer to split them across TP ranks instead of scoring every row on each rank.
`minimax_m3_indexer_cp` `config/attention.py:115`	`bool`	`False`	MiniMax M3 on ROCm with tensor_parallel_size > 1: score the sparse indexer context-parallel at decode. Each rank reads 1/TP of the index cache, scores every index head on it, and per-head top-k candidates are exchanged and merged, so the selected blocks match the default path. Requires a bf16 index cache (`indexer_kv_dtype` "auto" or "bf16").
`hisparse_config` `config/attention.py:122`	`HiSparseConfig \| None`	`None`	HiSparse host-resident KV configuration. Setting this enables experimental Model Runner V2-only HiSparse sparse-MLA decode hot-buffering. It is inferred with defaults when HiSparseConnector is configured (directly or via MultiConnector); set it explicitly only to tune its fields.
`use_non_causal` `config/attention.py:128`	`bool`	`False`	Whether to use non-causal (bidirectional) attention.
`sparse_mla_force_mqa` `config/attention.py:131`	`bool`	`False`	Force sparse MLA to use forward_mqa for all requests, including prefill. When False (default), pure prefill batches use forward_mha when implemented. Set to True to always use the MQA path.
`sparse_mla_prefill_workspace_factor` `config/attention.py:136`	`int \| None`	`Field(default=None, ge=2)`	Sparse MLA prefill workspace size, in multiples of `max_model_len`. FlashMLA sparse gathers a prefill step's context KV into this workspace and runs attention in more chunks when the step holds more. The sparse indexer's prefill K buffer is sized to fit in the same memory. Lower values return memory to the KV cache at long `max_model_len`; the minimum of 2 leaves room for a full-length request's …
`flex_attn_block_m` `config/attention.py:145`	`int \| None`	`None`	Triton kernel BLOCK_M tile size for flex attention. Must be a power of 2 >= 16. If None and VLLM_BATCH_INVARIANT=1, defaults to 16.
`flex_attn_block_n` `config/attention.py:150`	`int \| None`	`None`	Triton kernel BLOCK_N tile size for flex attention. Must be a power of 2 >= 16. If None and VLLM_BATCH_INVARIANT=1, defaults to 16.
`flex_attn_q_block_size` `config/attention.py:155`	`int \| None`	`None`	Logical Q block size for the flex attention block mask. Must be a power of 2 and divisible by flex_attn_block_m. If None, uses 16 for paged KV attention on PyTorch >= 2.9, and 128 for encoder-only attention or older PyTorch versions.
`flex_attn_kv_block_size` `config/attention.py:161`	`int \| None`	`None`	Logical KV block size for the flex attention block mask. Must be a power of 2 and divisible by flex_attn_block_n. If None, uses the KV cache block size for paged KV attention on PyTorch >= 2.9, and 128 for encoder-only attention or older PyTorch versions.

AuxOutputConfig#

config/aux_output.py:11, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`enable_return_routed_experts` `config/aux_output.py:14`	`bool`	`False`	Capture and return routed-experts auxiliary outputs.
`max_bytes` `config/aux_output.py:17`	`int \| None`	`Field(default=None, gt=0)`	LRU capacity, or ``None`` to derive it from the KV cache capacity.

CacheConfig#

config/cache.py:179, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`DEFAULT_BLOCK_SIZE` (ClassVar) `config/cache.py:182`	`ClassVar[int]`	`16`
`block_size` `config/cache.py:184`	`int`	`Field(default=None, gt=0)`	Size of a contiguous cache block in number of tokens. Accepts None (meaning "use default"). After construction, always int.
`user_specified_block_size` `config/cache.py:187`	`bool`	`field(default=False, init=False)`	Whether block_size was explicitly provided. Derived automatically.
`kv_cache_layout` `config/cache.py:189`	`str \| None`	`field(default=None, init=False)`	Resolved physical KV cache layout name (a ``KVCacheLayout`` member).
`prefix_match_unit` `config/cache.py:200`	`int \| None`	`Field(default=None, gt=0)`	The finest token boundary (in tokens) a prefix-cache hit can land on.
`gpu_memory_utilization` `config/cache.py:212`	`float`	`Field(default=0.92, gt=0, le=1)`	The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50% GPU memory utilization. If unspecified, will use the default value of 0.92. This is a per-instance limit, and only applies to the current vLLM instance. It does not matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM …
`cache_dtype` `config/cache.py:232`	`CacheDType \| str`	`"auto"`	Data type for kv cache storage. If "auto", will use model data type. An explicit float16 or bfloat16 must match the model data type. Custom dtypes can registered by platform backends via ``register_kv_cache_dtype`` are accepted (see vllm.config.cache). CUDA 11.8+ supports fp8 (=fp8_e4m3) and fp8_e5m2. ROCm (AMD GPU) supports fp8 (=fp8_e4m3). Intel Gaudi (HPU) supports fp8 (using fp8_inc). Some …
`is_attention_free` `config/cache.py:245`	`bool`	`False`	Whether the model is attention-free. This is primarily set in `ModelConfig` and that value should be manually duplicated here.
`num_gpu_blocks_override` `config/cache.py:248`	`int \| None`	`None`	Number of GPU blocks to use. This overrides the profiled `num_gpu_blocks` if specified. Does nothing if `None`. Used for testing preemption.
`sliding_window` `config/cache.py:251`	`int \| None`	`None`	Sliding window size for the KV cache. This is primarily set in `ModelConfig` and that value should be manually duplicated here.
`enable_prefix_caching` `config/cache.py:254`	`bool`	`True`	Whether to enable prefix caching.
`prefix_caching_hash_algo` `config/cache.py:256`	`PrefixCachingHashAlgo`	`"sha256"`	Set the hash algorithm for prefix caching:
`prefix_cache_retention_interval` `config/cache.py:272`	`int \| None`	`Field(default=0, ge=0)`	Token interval between retained sliding-window and Mamba prefix-cache checkpoints. ``0`` retains only semantic checkpoints, including the latest replay boundary and shared-prefix junctions. Positive values additionally retain periodic checkpoints at the specified interval, which must be a multiple of the scheduler block size. ``None`` retains checkpoints densely. Applies only to sliding-window …
`kv_cache_dtype_skip_layers` `config/cache.py:279`	`list[str]`	`field(default_factory=list)`	Layer patterns to skip KV cache quantization. Accepts layer indices (e.g., '0', '2', '4') or attention type names (e.g., 'sliding_window').
`mamba_page_size_padded` `config/cache.py:282`	`int \| None`	`None`	Optional override for mamba page size; used by hybrid mamba/attention models to ensure exact alignment with attention page size.
`skip_page_size_padded` `config/cache.py:285`	`int \| None`	`None`	Optional override for the page size of layers skipped from KV cache quantization (``--kv-cache-dtype-skip-layers``); set during block-size alignment so unquantized skip layers pad up to the quantized primary's page.
`mamba_block_size` `config/cache.py:290`	`int \| None`	`Field(default=None, gt=0)`	Size of a contiguous cache block in number of tokens for mamba cache. Can be set only when prefix caching is enabled.
`mamba_cache_dtype` `config/cache.py:293`	`MambaDType`	`"auto"`	The data type to use for the Mamba cache (both the conv as well as the ssm state). If set to 'auto', the data type will be inferred from the model config.
`mamba_ssm_cache_dtype` `config/cache.py:297`	`MambaDType`	`"auto"`	The data type to use for the Mamba cache (ssm state only, conv state will still be controlled by mamba_cache_dtype). If set to 'auto', the data type for the ssm state will be determined by mamba_cache_dtype.
`mamba_cache_mode` `config/cache.py:301`	`MambaCacheMode`	`"none"`	The cache strategy for Mamba layers, derived from `enable_prefix_caching`:
`enable_mamba_shared_prefix_checkpoint` `config/cache.py:309`	`bool`	`False`	Also register a Mamba "align" checkpoint at the shared-prefix junction -- where an EAGLE/MTP sibling was observed to resume -- instead of only at the prompt tail. Off by default; only takes effect with prefix caching enabled, EAGLE on the Mamba group, and a prefix match unit smaller than the Mamba block size.
`replayssm_buffer_len` `config/cache.py:315`	`int`	`Field(default=16, gt=0)`	ReplaySSM logical history length B for Mamba2. Triton uses B physical rows and FlashInfer uses B+T, where T is the target verification length. Kimi-K3 speculative decode does not use B. Default 16.
`use_replayssm` `config/cache.py:319`	`bool`	`False`	Use the ReplaySSM Mamba2 decode kernel: cache recent SSM inputs and skip the per-step full-state store, writing the checkpoint back only on flush. Requires the Triton or FlashInfer mamba backend; FlashInfer does not support prefix caching yet. Mamba2 speculative decode requires FlashInfer. With prefix caching, flushes are most efficient when mamba_block_size is a multiple of replayssm_buffer_len, …
`use_kda_recoverssm` `config/cache.py:326`	`bool`	`field(default=False, init=False)`	Whether Kimi-K3 KDA uses RecoverSSM speculative decode.
`num_gpu_blocks` `config/cache.py:330`	`int \| None`	`field(default=None, init=False)`	The number of blocks to allocate for GPU memory.
`num_cpu_blocks` `config/cache.py:332`	`int \| None`	`field(default=None, init=False)`	The number of blocks to allocate for CPU memory.
`simple_cpu_offload_num_blocks` `config/cache.py:335`	`int \| None`	`field(default=None, init=False)`	Worker-aligned offload block count for ``SimpleCPUOffloadConnector``.
`effective_attention_block_size` `config/cache.py:339`	`int \| None`	`field(default=None, init=False)`	Full-attention block size in tokens, including DCP, or None if unavailable.
`kv_cache_size_tokens` `config/cache.py:341`	`int \| None`	`field(default=None, init=False)`	Per-DP-engine KV cache capacity in tokens (group-aware). Uses group-aware capacity since num_gpu_blocks * block_size can be wrong for hybrid models where requests occupy multiple KV cache groups.
`kv_cache_max_concurrency` `config/cache.py:345`	`float \| None`	`field(default=None, init=False)`	Per-DP-engine maximum concurrency at max_model_len tokens.
`hash_block_size` `config/cache.py:351`	`int \| None`	`field(default=None, init=False)`	Tokens per prefix-cache block hash.
`cache_hit_alignment_tokens` `config/cache.py:353`	`int \| None`	`field(default=None, init=False)`	Token granularity at which prefix-cache hits land: `hash_block_size` with partial hash hits, otherwise the scheduler block size.
`kv_sharing_fast_prefill` `config/cache.py:357`	`bool`	`False`	In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254), some layers can skip tokens corresponding to prefill. This flag enables attention metadata for eligible layers to be overridden with metadata necessary for implementing this optimization in some models (e.g. Gemma3n)
`swa_bounded_replay` `config/cache.py:364`	`bool`	`True`	Keep the sliding-window KV of models that support it (DeepSeek-V4.1) out of prefix caching and rebuild it after a prefix hit by recomputing the hit's last window. The layers past the last KV-source layer then also run eager prefill steps on each request's trailing window only. Requires model runner V2.
`kv_cache_memory_bytes` `config/cache.py:371`	`int \| None`	`None`	Size of KV Cache per GPU in bytes. By default, this is set to None and vllm can automatically infer the kv cache size based on gpu_memory_utilization. However, users may want to manually specify the kv cache memory size. kv_cache_memory_bytes allows more fine-grain control of how much memory gets used when compared with using gpu_memory_utilization. Note that kv_cache_memory_bytes (when not-None) …
`kv_offloading_size` `config/cache.py:380`	`float \| None`	`None`	Size of the KV cache offloading buffer in GiB. When TP > 1, this is the total buffer size summed across all TP ranks. By default, this is set to None, which means no KV offloading is enabled. When set, vLLM will enable KV cache offloading to CPU using the kv_offloading_backend.
`kv_offloading_backend` `config/cache.py:386`	`KVOffloadingBackend`	`"native"`	The backend to use for KV cache offloading. Supported backends include 'native' (vLLM native CPU offloading), 'lmcache'. KV offloading is only activated when kv_offloading_size is set.
`_block_size_resolved` `config/cache.py:442`	`bool`	`field(default=False, init=False)`	Guard against pydantic re-running _apply_block_size_default.

PassConfig#

config/compilation.py:114, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`fuse_norm_quant` `config/compilation.py:128`	`bool`	`None`	Fuse the custom RMSNorm + quant ops.
`fuse_act_quant` `config/compilation.py:130`	`bool`	`None`	Fuse the custom SiluMul + quant ops.
`fuse_attn_quant` `config/compilation.py:132`	`bool`	`None`	Fuse the custom Attention and MLAAttention + quant ops.
`eliminate_noops` `config/compilation.py:134`	`bool`	`Field(default=True)`	Eliminate no-op ops.
`fuse_allreduce_rms` `config/compilation.py:136`	`bool`	`None`	Enable flashinfer allreduce fusion.
`enable_qk_norm_rope_fusion` `config/compilation.py:138`	`bool`	`None`	Enable fused Q/K RMSNorm + RoPE pass.
`fuse_rope_kvcache_cat_mla` `config/compilation.py:140`	`bool`	`None`	Enable fused MLA KV cache update with RoPE.
`fuse_act_padding` `config/compilation.py:144`	`bool`	`None`	Fuse the custom RMSNorm + padding ops.
`fuse_mla_dual_rms_norm` `config/compilation.py:146`	`bool`	`None`	Fuse paired q/kv RMS norms in MLA attention.
`fuse_rope_kvcache` `config/compilation.py:148`	`bool`	`None`	Fuse the QK rope + KV cache ops.
`fuse_qk_norm_rope_kvcache` `config/compilation.py:150`	`bool`	`Field(default=None)`	Fuse QK RMSNorm + RoPE/MRoPE + KV cache update into an AITER HIP kernel. Supersedes both enable_qk_norm_rope_fusion and fuse_rope_kvcache for layers that support it. Auto-enabled at O2+ on ROCm for models with QK-norm (e.g. Qwen3-MoE and Qwen3-VL-class architectures).
`rope_kvcache_fusion_max_token_num` `config/compilation.py:156`	`int`	`256`	The threshold for ROCm AITER RoPE+KVCache fusion e.g. for small batch decode. Larger batch sizes e.g. during prefill will use the unfused kernels. Also applies to the fused QK-Norm+RoPE/MRoPE+KVCache pass.
`fi_allreduce_fusion_max_size_mb` `config/compilation.py:162`	`float \| None`	`None`	The threshold of the communicated tensor sizes under which vllm should use flashinfer fused allreduce. Specified as a float in MB. Unspecified will fallback to default values which are compute capability and world size dependent. FI_ALLREDUCE_FUSION_MAX_SIZE_MB = { 90: { 2: 64, # 64MB 4: 2, # 2MB 8: 1, # 1MB }, 100: { 2: 64, # 64MB 4: 32, # 32MB 8: 1, # 1MB }, }, where key is the device capability

DynamicShapesConfig#

config/compilation.py:343, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`type` `config/compilation.py:346`	`DynamicShapesType`	`DynamicShapesType.BACKED`	Controls the type of dynamic shapes handling to use with torch.compile().
`evaluate_guards` `config/compilation.py:356`	`bool`	`False`	A debug mode to detect and fail if Dynamo ever specializes a dynamic shape by guarding on it. When True, dynamic shape guards are not dropped from dynamo. And a failure will be triggered if a recompilation ever happens due to that. This mode requires VLLM_USE_BYTECODE_HOOK to be 0. Enabling this allow observing the dynamic shapes guards in the tlparse artifacts also. When type is backed, …
`assume_32_bit_indexing` `config/compilation.py:368`	`bool`	`False`	whether all tensor sizes can use 32 bit indexing. `True` requires PyTorch 2.10+

CompilationConfig#

config/compilation.py:383, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`mode` `config/compilation.py:432`	`CompilationMode`	`None`	The compilation approach used for torch.compile-based compilation of the model.
`debug_dump_path` `config/compilation.py:446`	`Path \| None`	`None`	The path to dump the debug information.
`cache_dir` `config/compilation.py:448`	`str`	`""`	The directory to store the compiled graph, to accelerate Inductor compilation. By default, it will use model-related information to generate a cache directory.
`compile_cache_save_format` `config/compilation.py:452`	`Literal["binary", "unpacked"]`	`field(default_factory=lambda: envs.VLLM_COMPILE_CACHE_SAVE_FORMAT)`	Format for saving torch compile cache:
`backend` `config/compilation.py:461`	`str`	`""`	The backend for compilation. It needs to be a string:
`custom_ops` `config/compilation.py:480`	`list[str]`	`field(default_factory=list)`	Fine-grained control over which custom ops to enable/disable. Use 'all' to enable all, 'none' to disable all. Also specify a list of custom op names to enable (prefixed with a '+'), or disable (prefixed with a '-'). Examples:
`ir_enable_torch_wrap` `config/compilation.py:494`	`bool`	`None`	If True, enable vllm_ir torch custom op wrapping during the forward pass. When False, torch custom op wrapping is disabled, allowing Dynamo to trace the selected implementation directly or avoiding torch custom op overhead in eager mode. Defaults to True when using Inductor with vllm-compile (backend=="inductor" and mode == VLLM_COMPILE), False otherwise.
`splitting_ops` `config/compilation.py:502`	`list[str] \| None`	`None`	A list of ops to exclude from cudagraphs, used in piecewise compilation.
`compile_mm_encoder` `config/compilation.py:520`	`bool`	`False`	Whether or not to compile the multimodal encoder. Currently, this only works for `Qwen2_5_vl` and `mLLaMa4` models on selected platforms. It may also work for models loaded with the Transformers modeling backend if the encoder is compilable. Disabled by default until more models are supported/tested to work.
`cudagraph_mm_encoder` `config/compilation.py:528`	`bool`	`False`	Enable CUDA graph capture for multimodal encoder (ViT). When enabled, captures full encoder forward as CUDA graph for each token budget level.
`encoder_cudagraph_token_budgets` `config/compilation.py:533`	`list[int]`	`field(default_factory=list)`	Token budget levels for encoder CUDA graph capture. Each budget defines a fixed token capacity. At runtime, images are greedy-packed into the smallest fitting budget and the corresponding CUDA graph is replayed. If empty (default), auto-inferred from model architecture as power-of-2 levels from the model's estimated min budget to max budget. User-provided values override auto-inference. Example: …
`encoder_cudagraph_max_vision_items_per_batch` `config/compilation.py:542`	`int`	`0`	Maximum number of images/videos per batch for encoder CUDA graph capture. Determines the fixed batch size used during graph capture. If 0 (default), auto-inferred as max_budget // min_budget from the model's budget range. User-provided positive value overrides auto-inference.
`encoder_cudagraph_max_frames_per_batch` `config/compilation.py:549`	`int \| None`	`None`	Maximum total video frames per batch for encoder CUDA graph capture. Controls the cu_seqlens buffer size (one entry per attention sequence, i.e. one per video frame). If None (default), auto-inferred as encoder_cudagraph_max_vision_items_per_batch * max_frames_per_video (model-specific value according to processing_info). Positive value overrides auto-inference and applies to all budget levels. …
`cudagraph_decoder_replay` `config/compilation.py:559`	`bool`	`True`	Run the decoder replay layers of YOCO models (e.g. DeepSeek-V4.1) in CUDA graphs of their own, and trim them in PIECEWISE graph steps. Requires breakable PIECEWISE graphs; off under LoRA, prompt embeddings and non-first PP ranks.
`decoder_replay_cudagraph_capture_sizes` `config/compilation.py:565`	`list[int]`	`field(default_factory=list)`	Decoder replay CUDA graph sizes for YOCO models (e.g. DeepSeek-V4.1), at most max_num_batched_tokens. If empty: multiples of sliding_window up to min(max_cudagraph_capture_size, max_num_seqs * sliding_window), coarser past 16 * sliding_window. Larger replay batches run eagerly.
`decoder_replay_trim_threshold` `config/compilation.py:571`	`int`	`Field(default=768, ge=0)`	For YOCO models (e.g. DeepSeek-V4.1), PIECEWISE graphs with at least this many padded tokens trim the decoder replay batch; must exceed the replay window (128), below which nothing trims. Independent of the replay graph capture sizes; eager steps still trim.
`compile_sizes` `config/compilation.py:578`	`list[int \| str] \| None`	`None`	Sizes to compile for inductor. In addition to integers, it also supports "cudagraph_capture_sizes" to specify the sizes for cudagraph capture.
`compile_ranges_endpoints` `config/compilation.py:583`	`list[int] \| None`	`None`	Endpoints for Inductor compile ranges. The compile ranges are [1, endpoints[0]], [endpoints[0] + 1, endpoints[1]], ..., [endpoints[-1] + 1, max_num_batched_tokens]. Compile sizes are also used single element ranges, the range is represented as [compile_sizes[i], compile_sizes[i]].
`inductor_compile_config` `config/compilation.py:598`	`dict`	`field(default_factory=dict)`	Additional configurations for inductor. - None: use default configurations.
`inductor_passes` `config/compilation.py:602`	`dict[str, str]`	`field(default_factory=dict)`	Additional passes for inductor. It is a dictionary from pass name to pass function qualified name. We use function name because the config uses JSON format. If we pass the config from Python, functions can also be passed directly via Python object constructor, e.g. `CompilationConfig(inductor_passes={"a": func})`.
`cudagraph_mode` `config/compilation.py:610`	`CUDAGraphMode`	`None`	The mode of the cudagraph:
`cudagraph_num_of_warmups` `config/compilation.py:646`	`int`	`0`	Number of warmup runs for cudagraph. It means the first several runs will be treated as warmup runs. Only after that, the execution will be recorded, and the recorded cudagraph will be used for subsequent runs.
`cudagraph_capture_sizes` `config/compilation.py:651`	`list[int]`	`None`	Sizes to capture cudagraph. - None (default): capture sizes are inferred from vllm config. - list[int]: capture sizes are specified as given.
`cudagraph_copy_inputs` `config/compilation.py:655`	`bool`	`False`	Whether to copy input tensors for cudagraph. If the caller can guarantee that the same input buffers are always used, it can set this to False. Otherwise, it should set this to True, and the compiler will copy the input to an internally managed buffer. Default is False. Note that this flag is only effective when cudagraph_mode is PIECEWISE.
`cudagraph_specialize_lora` `config/compilation.py:663`	`bool`	`True`	Whether to create separate cuda graphs for cases with and without active LoRA adapters. When set to False, the LoRA-enabled cuda graph will be used for all cases, incurring the overhead of running LoRA ops even when no adapters are active. Setting this to True will remove this overhead at the cost of increased startup time and slightly higher memory usage. When `enable_lora` is False, this option …
`use_inductor_graph_partition` `config/compilation.py:672`	`bool`	`None`	Use inductor graph partition to split the graph at cudagraph_unsafe ops. This partition happens at inductor codegen time after all passes and fusions are finished. It generates a single `call` function which wraps cudagraph-safe ops into partition functions and leave cudagraph-unsafe ops outside the partition functions. For a graph with N cudagraph-unsafe ops (e.g., Attention), there would be N+1 …
`pass_config` `config/compilation.py:692`	`PassConfig`	`field(default_factory=PassConfig)`	Custom inductor passes, see PassConfig for more details
`max_cudagraph_capture_size` `config/compilation.py:695`	`int`	`None`	The maximum cudagraph capture size.
`dynamic_shapes_config` `config/compilation.py:713`	`DynamicShapesConfig`	`field(default_factory=DynamicShapesConfig)`	Configuration for dynamic shapes options
`local_cache_dir` `config/compilation.py:718`	`str`	`field(default=None, init=False)`	local cache dir for each rank
`fast_moe_cold_start` `config/compilation.py:721`	`bool \| None`	`None`	Optimization for fast MOE cold start.
`enabled_custom_ops` `config/compilation.py:746`	`Counter[str]`	`field(default_factory=Counter, init=False)`	custom ops that are enabled
`disabled_custom_ops` `config/compilation.py:748`	`Counter[str]`	`field(default_factory=Counter, init=False)`	custom ops that are disabled
`traced_files` `config/compilation.py:750`	`set[str]`	`field(default_factory=set, init=False)`	files that are traced for compilation
`compilation_time` `config/compilation.py:752`	`float`	`field(default=0.0, init=False)`	time taken for compilation
`encoder_compilation_time` `config/compilation.py:754`	`float`	`field(default=0.0, init=False)`	time taken for multimodal encoder compilation
`static_forward_context` `config/compilation.py:757`	`dict[str, Any]`	`field(default_factory=dict, init=False)`	Per-model forward context Map from layer name to layer objects that need to be accessed outside model code, e.g., Attention, FusedMOE when dp_size>1.
`static_all_moe_layers` `config/compilation.py:762`	`list[str]`	`field(default_factory=list, init=False)`	The names of all the MOE layers in the model
`_attention_ops` (ClassVar) `config/compilation.py:768`	`ClassVar[list[str]]`	`["vllm::unified_attention_with_output", "vllm::unified_mla_attention_with_output", "vllm::mamba_mixer2", "vllm::mamba_mixer", "vllm::short_c …`

DeviceConfig#

config/device.py:17, decorator @config(config=ConfigDict(arbitrary_types_allowed=True)).

Field	Type	Default	Docstring (first paragraph)
`device` `config/device.py:20`	`SkipValidation[Device \| torch.device \| None]`	`"auto"`	Device type for vLLM execution. This parameter is deprecated and will be removed in a future release. It will now be set automatically based on the current platform.
`device_type` `config/device.py:26`	`str`	`field(init=False)`	Device type from the current platform. This is set in `__post_init__`.

DiffusionConfig#

config/diffusion.py:11, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`canvas_length` `config/diffusion.py:20`	`int`	`Field(default=None, gt=0)`	Length of the denoising canvas (block). Also determines the number of speculative tokens scheduled per step.
`max_denoising_steps` `config/diffusion.py:24`	`int \| None`	`None`	Maximum number of denoising iterations per canvas block. If not set, read from the model's generation_config.json.

EncoderCacheManagerConfig#

config/ec_manager_config.py:13, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`encoder_cache_manager_cls` `config/ec_manager_config.py:14`	`str \| None`	`None`	Fully qualified class name of the custom encoder cache manager.
`manager_config` `config/ec_manager_config.py:17`	`dict[str, Any]`	`field(default_factory=dict)`	Opaque configuration interpreted by the custom cache manager.

ECTransferConfig#

config/ec_transfer.py:16, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`ec_connector` `config/ec_transfer.py:19`	`str \| None`	`None`	The EC connector for vLLM to transmit EC caches between vLLM instances.
`engine_id` `config/ec_transfer.py:30`	`str \| None`	`None`	The engine id for EC transfers.
`ec_buffer_device` `config/ec_transfer.py:33`	`str \| None`	`"cuda"`	The device used by ec connector to buffer the EC cache. Currently only support 'cuda'.
`ec_buffer_size` `config/ec_transfer.py:37`	`float`	`1e9`	The buffer size for TorchDistributedConnector. Measured in number of bytes. Recommended value: 1e9 (about 1GB).
`ec_role` `config/ec_transfer.py:41`	`ECRole \| None`	`None`	Whether this vLLM instance produces, consumes EC cache, or both. Choices are 'ec_producer', 'ec_consumer', 'ec_both'.
`ec_ip` `config/ec_transfer.py:45`	`str`	`"127.0.0.1"`	The EC connector ip, used to build distributed connection.
`ec_port` `config/ec_transfer.py:48`	`int`	`14579`	The EC connector port, used to build distributed connection.
`ec_connector_extra_config` `config/ec_transfer.py:51`	`dict[str, Any]`	`field(default_factory=dict)`	any extra config that the connector may need.
`ec_connector_module_path` `config/ec_transfer.py:54`	`str \| None`	`None`	The Python module path to dynamically load the EC connector from. Only supported in V1.
`mm_encoder_output_widths` `config/ec_transfer.py:58`	`dict[str, int]`	`field(default_factory=dict, init=False)`	Per-token width of the encoder output for each modality, measured by EC producers at startup. It can exceed the LM embedding width (e.g. DeepStack).

EngramConfig#

config/engram.py:34, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`cpu_offload` `config/engram.py:37`	`bool`	`True`	Store embedding weights in pinned CPU memory for UVA lookup.
`embedding_across_dp` `config/engram.py:40`	`bool`	`False`	Shard embeddings across TP and all DP ranks when enabled. Otherwise, each DP rank has a separate TP-sharded embedding replica.
`dp_shared_memory` `config/engram.py:44`	`bool \| None`	`None`	Share CPU-offloaded embedding weights between co-located DP replicas. Each node stores one copy of every TP shard, reducing host memory without per-step Engram DP collectives. Requires sufficient /dev/shm capacity and a shared IPC namespace. Defaults to enabled whenever the other settings allow it, falling back to per-replica tables when DP replicas are not co-located on one node or /dev/shm …
`use_thp` `config/engram.py:52`	`bool`	`False`	Back private CPU-offloaded tables with transparent huge pages (best effort, falls back to ordinary pinned pages). Prefaulting the tables at startup takes longer. Requires cpu_offload without dp_shared_memory.

FaultToleranceConfig#

config/fault_tolerance.py:9, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`engine_recovery_timeout_sec` `config/fault_tolerance.py:12`	`int`	`120`	Timeout (in seconds) to wait for error handling instructions before raising an exception. If the EngineCore encounters an error, it waits up to this many seconds for vLLM to receive instructions on how to handle the error and then recover from the fault. If vLLM does not recover during this time, the original error is raised.

IrOpPriorityConfig#

config/kernel.py:21, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`rms_norm` `config/kernel.py:31`	`list[str]`	`Field(default_factory=list)`	Priority list for vllm.ir.ops.rms_norm
`fused_add_rms_norm` `config/kernel.py:34`	`list[str]`	`Field(default_factory=list)`	Priority list for vllm.ir.ops.fused_add_rms_norm
`gelu_and_mul_sparse` `config/kernel.py:37`	`list[str]`	`Field(default_factory=list)`	Priority list for vllm.ir.ops.gelu_and_mul_sparse

KernelConfig#

config/kernel.py:271, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`ir_op_priority` `config/kernel.py:274`	`IrOpPriorityConfig`	`Field(default_factory=IrOpPriorityConfig)`	vLLM IR op priority for dispatching/lowering during the forward pass. Platform defaults appended automatically during VllmConfig.__post_init__.
`enable_flashinfer_autotune` `config/kernel.py:280`	`bool`	`None`	Benchmark FlashInfer kernel implementations during warmup and use the fastest. Enabled by default except at `-O0`; disable for faster startup.
`enable_rocm_segmented_attn_autotune` `config/kernel.py:284`	`bool`	`False`	If True, autotune ROCm segmented attention during kernel warmup on RDNA GPUs.
`enable_cutedsl_warmup` `config/kernel.py:290`	`bool`	`True`	Deprecated: run legacy CuTeDSL warmup providers.
`enable_jit_warmup` `config/kernel.py:293`	`bool`	`True`	If True, run JIT compile warmup during kernel warmup.
`monokernel` `config/kernel.py:296`	`bool`	`False`	If True, run the eligible layers of decode steps on the model's MonoKernel: a persistent kernel that runs most or all of one decoder layer in a single launch (DeepSeek-V4.1 on MI355X). A deployment that cannot run it is refused at startup.
`moe_backend` `config/kernel.py:302`	`MoEBackend`	`"auto"`	Backend for MoE expert computation kernels. Available options:
`sparse_indexer_topk_backend` `config/kernel.py:343`	`SparseIndexerTopkBackend`	`"auto"`	Backend for the DSA sparse indexer decode top-k kernel. Available options:
`linear_backend` `config/kernel.py:360`	`LinearBackend`	`"auto"`	Backend for linear layer GEMM kernels. Available options:
`linear_backend_per_quant` `config/kernel.py:389`	`dict[str, LinearBackend] \| None`	`Field(default=None, min_length=1)`	Backend overrides keyed by linear quantization scheme. Overrides take precedence over ``linear_backend``; for example, ``{"nvfp4_w4a16": "humming"}``.

KVEventsConfig#

config/kv_events.py:11, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`enable_kv_cache_events` `config/kv_events.py:14`	`bool`	`False`	If True, enable KV cache events for tracking block storage and removal. Events can be published externally by zmq using the event publisher config.
`publisher` `config/kv_events.py:19`	`Literal["null", "zmq"]`	`None`	The publisher to use for publishing kv events. Can be "null", "zmq".
`endpoint` `config/kv_events.py:23`	`str`	`"tcp://*:5557"`	The zmq endpoint to use for publishing kv events.
`replay_endpoint` `config/kv_events.py:27`	`str \| None`	`None`	The zmq endpoint to use for replaying kv events.
`buffer_steps` `config/kv_events.py:31`	`int`	`10_000`	The number of steps to cache for replay endpoint. Will only save events from the last N steps for the replay endpoint.
`hwm` `config/kv_events.py:36`	`int`	`100_000`	The zmq high water mark for the event publisher. After queueing N events, events will start dropping if the consumer is not keeping up.
`max_queue_size` `config/kv_events.py:41`	`int`	`100_000`	The maximum number of events to queue while waiting for publishing.
`topic` `config/kv_events.py:45`	`str`	`""`	The topic to use for the event publisher. Consumers can subscribe to this topic to receive events.

KVTransferConfig#

config/kv_transfer.py:57, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`kv_connector` `config/kv_transfer.py:60`	`str \| None`	`None`	The KV connector for vLLM to transmit KV caches between vLLM instances.
`engine_id` `config/kv_transfer.py:64`	`str \| None`	`None`	The engine id for KV transfers.
`kv_buffer_device` `config/kv_transfer.py:67`	`str`	`field(default_factory=kv_buffer_device_default_factory)`	The device used by kv connector to buffer the KV cache. Choices are 'cuda', 'cpu' and 'xpu'.
`kv_role` `config/kv_transfer.py:71`	`KVRole \| None`	`None`	Whether this vLLM instance produces, consumes KV cache, or both. Choices are 'kv_producer', 'kv_consumer', and 'kv_both'.
`kv_port` `config/kv_transfer.py:75`	`int`	`14579`	The KV connector port, used to build distributed connection.
`kv_connector_extra_config` `config/kv_transfer.py:78`	`dict[str, Any]`	`field(default_factory=dict)`	any extra config that the connector may need.
`kv_connector_module_path` `config/kv_transfer.py:81`	`str \| None`	`None`	The Python module path to dynamically load the KV connector from. Only supported in V1.
`enable_permute_local_kv` `config/kv_transfer.py:85`	`bool`	`False`	Experiment feature flag to enable HND to NHD KV Transfer
`kv_load_failure_policy` `config/kv_transfer.py:88`	`Literal["recompute", "fail"]`	`"fail"`	Policy for handling KV cache load failures. 'recompute': reschedule the request to recompute failed blocks 'fail': immediately fail the request with an error finish reason (default)

LoadConfig#

config/load.py:27, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`load_format` `config/load.py:30`	`str \| LoadFormats`	`"auto"`	The format of the model weights to load.
`download_dir` `config/load.py:62`	`str \| None`	`None`	Directory to download and load the weights, default to the default cache directory of Hugging Face.
`safetensors_load_strategy` `config/load.py:65`	`SafetensorsLoadStrategy \| None`	`None`	Specifies the loading strategy for safetensors weights.
`safetensors_prefetch_num_threads` `config/load.py:87`	`int`	`Field(default=DEFAULT_SAFETENSORS_PREFETCH_NUM_THREADS, ge=1)`	Number of worker threads used to prefetch safetensors checkpoint files into the OS page cache when safetensors prefetching is enabled.
`safetensors_prefetch_block_size` `config/load.py:92`	`int`	`Field(default=DEFAULT_SAFETENSORS_PREFETCH_BLOCK_SIZE, ge=1)`	Read size in bytes for each safetensors checkpoint file prefetch.
`model_loader_extra_config` `config/load.py:96`	`dict \| TensorizerConfig`	`Field(default_factory=dict)`	Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format.
`device` `config/load.py:99`	`str \| None`	`None`	Device to which model weights will be loaded, default to device_config.device
`ignore_patterns` `config/load.py:102`	`list[str] \| str`	`Field(default_factory=lambda: ["original/*/"])`	The list of patterns to ignore when loading the model. Default to "original/*/" to avoid repeated loading of llama's checkpoints.
`use_tqdm_on_load` `config/load.py:105`	`bool`	`True`	Whether to enable tqdm for showing progress bar when loading model weights.
`pt_load_map_location` `config/load.py:108`	`str \| dict[str, str]`	`"cpu"`	The map location for loading pytorch checkpoint, to support loading checkpoints can only be loaded on certain devices like "cuda", this is equivalent to `{"": "cuda"}`. Another supported format is mapping from different devices like from GPU 1 to GPU 0: `{"cuda:1": "cuda:0"}`. Note that when passed from command line, the strings in dictionary need to be double quoted for json parsing. For more …

LoggingConfig#

config/logging.py:16, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`log_level` `config/logging.py:26`	`LogLevel`	`Field(default_factory=lambda: cast(LogLevel, envs.VLLM_LOGGING_LEVEL))`	Log level used when no custom logging configuration is provided.
`formatter` `config/logging.py:31`	`LogFormatter`	`"text"`	Formatter used by the built-in logging configuration.
`configure_logging` `config/logging.py:34`	`bool`	`Field(default_factory=lambda: envs.VLLM_CONFIGURE_LOGGING)`	Whether to apply a Python logging configuration.
`pylogging_config_file` `config/logging.py:42`	`str \| None`	`Field(default_factory=lambda: envs.VLLM_LOGGING_CONFIG_PATH)`	Path to a Python logging JSON file using [``dictConfig`` schema](https://docs.python.org/3/library/logging.config.html#configuration-file-format).

LoRAConfig#

config/lora.py:30, decorator @config(config=ConfigDict(arbitrary_types_allowed=True)).

Field	Type	Default	Docstring (first paragraph)
`max_lora_rank` `config/lora.py:33`	`MaxLoRARanks`	`16`	Max LoRA rank.
`max_loras` `config/lora.py:35`	`int`	`Field(default=1, ge=1)`	Max number of LoRAs in a single batch.
`fully_sharded_loras` `config/lora.py:37`	`bool`	`False`	By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster.
`max_cpu_loras` `config/lora.py:42`	`int \| None`	`None`	Maximum number of LoRAs to store in CPU memory. Must be >= than `max_loras`.
`lora_dtype` `config/lora.py:45`	`torch.dtype \| LoRADType`	`"auto"`	Data type for LoRA. If auto, will default to base model dtype.
`target_modules` `config/lora.py:47`	`list[str] \| None`	`None`	Restrict LoRA to specific module suffixes (e.g., ["o_proj", "qkv_proj"]). If None, all supported LoRA modules are used. This allows deployment-time control over which modules have LoRA applied, useful for performance tuning.
`default_mm_loras` `config/lora.py:51`	`dict[str, str] \| None`	`None`	Dictionary mapping specific modalities to LoRA model paths; this field is only applicable to multimodal models and should be leveraged when a model always expects a LoRA to be active when a given modality is present. Note that currently, if a request provides multiple additional modalities, each of which have their own LoRA, we do NOT apply default_mm_loras because we currently only support one …
`enable_tower_connector_lora` `config/lora.py:61`	`bool`	`False`	If `True`, LoRA support for the tower (vision encoder) and connector of multimodal models will be enabled. This is an experimental feature and currently only supports some MM models such as the Qwen VL series. The default is False.
`specialize_active_lora` `config/lora.py:66`	`bool`	`False`	Whether to construct lora kernel grid by the number of active LoRA adapters. When set to True, separate cuda graphs will be captured for different counts of active LoRAs (powers of 2 up to max_loras), which can improve performance for variable LoRA usage patterns at the cost of increased startup time and memory usage. Only takes effect when cudagraph_specialize_lora is True.
`enable_mixed_moe_lora_format` `config/lora.py:73`	`bool`	`False`	If True, force the engine to use the universal 2D MoE LoRA wrapper (`FusedMoEWithLoRA`) regardless of the model's `is_3d_moe_weight` flag, so that 2D-format and 3D-format MoE LoRA adapters can be served in the same deployment. Only meaningful for MoE models; ignored otherwise. Default False keeps the existing model-driven behavior.
`enable_moe_shared_loras` `config/lora.py:79`	`bool`	`False`	If True, load MoE expert adapters in the "shared-outer" layout, where the gate/up (`w1`/`w3`) lora_A and the down (`w2`) lora_B are shared across all experts (stored once with expert-dim 1) instead of per-expert. The shared factors are broadcast to the expert count at kernel time. 3D-weight MoE models use `FusedMoEWithLoRA` for this layout. Only meaningful for MoE models whose adapters use this …
`max_lora_cls_labels` `config/lora.py:86`	`int \| None`	`Field(default=None, ge=1)`	Maximum output size for LoRA classification heads. Defaults to the base classification head size.

MambaConfig#

config/mamba.py:37, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`backend` `config/mamba.py:40`	`MambaBackendEnum`	`MambaBackendEnum.TRITON`	Mamba SSU backend to use.
`enable_stochastic_rounding` `config/mamba.py:43`	`bool`	`False`	Enable stochastic rounding when writing SSM state to fp16 cache. Uses random bits to unbias the rounding error, which can improve numerical stability for long sequences.
`stochastic_rounding_philox_rounds` `config/mamba.py:47`	`int`	`0`	Number of Philox PRNG rounds for stochastic rounding random number generation. 0 uses the Triton default. Higher values improve randomness quality at the cost of compute.
`ssu_algorithm` `config/mamba.py:52`	`MambaSSUAlgorithm \| None`	`None`	Selective state update algorithm to use with the FlashInfer backend. None defaults to FlashInfer's "auto" algorithm. Forced algorithms must be supported by FlashInfer for the active GPU, state dtype, and decoding mode.

ModelConfig#

config/model.py:152, decorator @config(config=ConfigDict(arbitrary_types_allowed=True)).

Field	Type	Default	Docstring (first paragraph)
`model` `config/model.py:155`	`str`	`"Qwen/Qwen3-0.6B"`	Name or path of the Hugging Face model to use. It is also used as the content for `model_name` tag in metrics output when `served_model_name` is not specified.
`model_weights` `config/model.py:159`	`str`	`""`	Original model weights path. Used when the model is pulled from object storage (e.g., RunAI) to preserve the original URI while `model` points to the local directory.
`runner` `config/model.py:163`	`RunnerOption`	`"auto"`	The type of model runner to use. Each vLLM instance only supports one model runner, even if the same model can be used for multiple types.
`convert` `config/model.py:166`	`ConvertOption`	`"auto"`	Convert the model using adapters defined in [vllm.model_executor.models.adapters][]. The most common use case is to adapt a text generation model to be used for pooling tasks.
`tokenizer` `config/model.py:170`	`str`	`None`	Name or path of the Hugging Face tokenizer to use. If unspecified, model name or path will be used.
`tokenizer_mode` `config/model.py:173`	`TokenizerMode \| str`	`"auto"`	Tokenizer mode:
`trust_remote_code` `config/model.py:197`	`bool`	`False`	Trust remote code (e.g., from HuggingFace) when downloading the model and tokenizer.
`dtype` `config/model.py:200`	`ModelDType \| torch.dtype`	`"auto"`	Data type for model weights and activations:
`seed` `config/model.py:210`	`int`	`0`	Random seed for reproducibility.
`hf_config` `config/model.py:216`	`PreTrainedConfig`	`field(init=False)`	The Hugging Face config of the model.
`hf_text_config` `config/model.py:218`	`PreTrainedConfig`	`field(init=False)`	The Hugging Face config of the text model (same as hf_config for text models).
`is_submodel_config` `config/model.py:220`	`bool`	`field(default=False, init=False)`	Whether this is a submodule view derived by `VllmConfig.with_hf_config` (e.g. a multimodal model's text stack). Its architecture list is empty, so deployment-level validation must not run against it.
`word_embeddings_untied_by_checkpoint` `config/model.py:224`	`bool`	`field(default=False, init=False)`	Whether `tie_word_embeddings` was overridden to `False` because the checkpoint contains an `lm_head` of its own. The two may still turn out to be identical, in which case they are re-tied once the weights have been loaded.
`hf_config_path` `config/model.py:228`	`str \| None`	`None`	Name or path of the Hugging Face config to use. If unspecified, model name or path will be used.
`allowed_local_media_path` `config/model.py:231`	`str`	`""`	Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments.
`allowed_media_domains` `config/model.py:235`	`list[str] \| None`	`None`	If set, only media URLs that belong to this domain can be used for multi-modal inputs.
`revision` `config/model.py:238`	`str \| None`	`None`	The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.
`code_revision` `config/model.py:241`	`str \| None`	`None`	The specific revision to use for the model code on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.
`tokenizer_revision` `config/model.py:245`	`str \| None`	`None`	The specific revision to use for the tokenizer on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.
`max_model_len` `config/model.py:249`	`int`	`Field(default=None, ge=-1)`	Model context length (prompt and output). If unspecified, will be automatically derived from the model config.
`spec_target_max_model_len` `config/model.py:262`	`int \| None`	`None`	Specify the maximum length for spec decoding draft models.
`quantization` `config/model.py:264`	`QuantizationMethods \| str \| None`	`None`	Method used to quantize the weights. If `None`, we first check the `quantization_config` attribute in the model config file. If that is `None`, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights.
`quantization_config` `config/model.py:269`	`dict[str, Any] \| QuantizationConfigArgs \| None`	`None`	User-facing quantization configuration. Carries per-layer-kind specs (linear, moe) and ignore patterns; see :class:`QuantizationConfigArgs`. Auto-populated from the matching online shorthand when `quantization` is one of the values in `ONLINE_QUANT_SHORTHAND_NAMES`.
`allow_deprecated_quantization` `config/model.py:274`	`bool`	`False`	Whether to allow deprecated quantization methods.
`enforce_eager` `config/model.py:276`	`bool`	`False`	Whether to always use eager-mode PyTorch. If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid for maximal performance and flexibility.
`return_sampling_mask` `config/model.py:284`	`bool`	`False`	Whether to return the post-processing token support for each sample.
`max_logprobs` `config/model.py:286`	`int`	`Field(default=20, ge=-1)`	Maximum number of log probabilities to return when `logprobs` is specified in `SamplingParams`. The default value comes the default for the OpenAI Chat Completions API. -1 means no cap, i.e. all (output_length * vocab_size) logprobs are allowed to be returned and it may cause OOM.
`logprobs_mode` `config/model.py:291`	`LogprobsMode`	`"raw_logprobs"`	Indicates the content returned in the logprobs and prompt_logprobs. Supported mode: 1) raw_logprobs, 2) processed_logprobs, 3) raw_logits, 4) processed_logits. Raw means the values before applying any logit processors, like bad words. Processed means the values after applying all processors, including temperature and top_k/top_p. Note: for prompt_logprobs, processed_* and raw_* yield identical …
`use_fp64_gumbel` `config/model.py:301`	`bool`	`False`	Whether to use FP64 (instead of FP32) random noise for Gumbel-max and equivalent exponential-race sampling. FP64 preserves lower-tail sampling events that fp32 uniform/exponential draws can truncate, at the cost of significantly lower throughput on most GPUs.
`enable_trace_replay` `config/model.py:306`	`bool`	`False`	Whether to allow requests to set `SamplingParams.trace_decode_token_ids`, which forces decoding to follow a predetermined token sequence while still computing real logprobs. Reserved for debugging and RL workflows: enabling it reserves a per-request trace buffer, so it is off by default.
`disable_sliding_window` `config/model.py:312`	`bool`	`False`	Whether to disable sliding window. If True, we will disable the sliding window functionality of the model, capping to sliding window size. If the model does not support sliding window, this argument is ignored.
`disable_cascade_attn` `config/model.py:316`	`bool`	`True`	Deprecated: cascade attention has been removed, so this has no effect. It will be removed in v0.33.0.
`skip_tokenizer_init` `config/model.py:319`	`bool`	`False`	Skip initialization of tokenizer and detokenizer. Expects valid `prompt_token_ids` and `None` for prompt from the input. The generated output will contain token ids.
`enable_prompt_embeds` `config/model.py:323`	`bool`	`False`	If `True`, enables passing text embeddings as inputs via the `prompt_embeds` key.
`served_model_name` `config/model.py:329`	`str \| list[str] \| None`	`None`	The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the `--model` argument. Noted that this name(s) will also be used in `model_name` tag content of prometheus metrics, if multiple names …
`config_format` `config/model.py:337`	`str \| ConfigFormat`	`"auto"`	The format of the model config to load:
`hf_token` `config/model.py:344`	`bool \| str \| None`	`None`	The token to use as HTTP bearer authorization for remote files . If `True`, will use the token generated when running `hf auth login` (stored in `~/.cache/huggingface/token`).
`hf_overrides` `config/model.py:348`	`HfOverrides`	`field(default_factory=dict)`	If a dictionary, contains arguments to be forwarded to the Hugging Face config. If a callable, it is called to update the HuggingFace config.
`model_class_overrides` `config/model.py:351`	`dict[str, str]`	`field(default_factory=dict)`	Override the model class used for one or more architectures, mapping the architecture name to a `"module:class"` target (the same format accepted by `ModelRegistry.register_model`). This registers the target class at runtime, e.g. `{"GlmMoeDsaForCausalLM": "vllm.models.deepseek_v32.nvidia.model:DeepseekV32ForCausalLM"}`. This argument is for development and debugging purposes only.
`generation_config` `config/model.py:358`	`str`	`"auto"`	The folder path to the generation config. Defaults to `"auto"`, the generation config will be loaded from model path. If set to `"vllm"`, no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If `max_new_tokens` is specified in generation config, then it sets a server-wide limit on the number of …
`override_generation_config` `config/model.py:365`	`dict[str, Any]`	`field(default_factory=dict)`	Overrides or sets generation config. e.g. `{"temperature": 0.5}`. If used with `--generation-config auto`, the override parameters will be merged with the default config from the model. If used with `--generation-config vllm`, only the override parameters are used.
`enable_sleep_mode` `config/model.py:370`	`bool`	`False`	Enable sleep mode for the engine (only cuda and hip platforms are supported).
`sleep_preserve_parameter_names` `config/model.py:373`	`list[str]`	`field(default_factory=list)`	Parameter-name globs to preserve across level-2 sleep. Use target globs for frozen Engram table weights and scales omitted by the trainer (e.g. ``.engram.embed_tokens.weight``). Use ``draft.<name>`` for draft-private parameters, or ``draft.*`` for all. The sender must omit these parameters; loaders must preserve their storage.
`sleep_mode_backend` `config/model.py:379`	`str`	`"cumem"`	Mechanism used to free and restore GPU state for sleep mode. ``"cumem"`` (default) uses the built-in ``CuMemAllocator`` and is behavior-compatible with prior releases. Additional backends (CUDA checkpoint, CRIU, durable snapshot) may be registered in-tree or by plugins (RFC #34303).
`sleep_mode_offload_cudagraph` `config/model.py:384`	`bool`	`False`	Back up CUDA graph memory to CPU during sleep, restored in place on wake. Takes effect with enable_sleep_mode, the cumem backend and CUDA graphs.
`enable_nccl_comm_suspend` `config/model.py:387`	`bool`	`False`	Enable releasing NCCL communicator memory during sleep mode (``ncclCommSuspend``/``ncclCommResume``). Experimental; when disabled (the default) sleep still releases weights/KV-cache memory as before.
`enable_cumem_allocator` `config/model.py:391`	`bool`	`False`	Enable the custom cumem allocator to leverage advanced GPU memory allocation features such as multi-node NVLink support.
`model_impl` `config/model.py:398`	`str \| ModelImpl`	`"auto"`	Which implementation of the model to use:
`logits_processors` `config/model.py:407`	`list[str \| type[LogitsProcessor]] \| None`	`None`	One or more logits processors' fully-qualified class names or class definitions
`io_processor_plugin` `config/model.py:410`	`str \| None`	`None`	IOProcessor plugin name to load at model startup
`renderer_num_workers` `config/model.py:412`	`int`	`1`	Number of worker threads in the renderer thread pool. The pool is consumed by the async renderer path (e.g. the OpenAI-compatible API server started by `vllm serve`) to parallelize tokenization, chat template rendering, and multimodal preprocessing across concurrent requests.
`pooler_config` `config/model.py:424`	`PoolerConfig \| None`	`None`	Pooler config which controls the behaviour of output pooling in pooling models.
`multimodal_config` `config/model.py:429`	`MultiModalConfig \| None`	`None`	Configuration for multimodal model. If `None`, this will be inferred from the architecture of `self.model`.
`language_model_only` (InitVar) `config/model.py:432`	`InitVar[bool]`	`False`
`limit_mm_per_prompt` (InitVar) `config/model.py:433`	`InitVar[dict[str, int \| dict[str, int]] \| None]`	`None`
`enable_mm_embeds` (InitVar) `config/model.py:434`	`InitVar[bool \| None]`	`None`
`media_io_kwargs` (InitVar) `config/model.py:435`	`InitVar[dict[str, dict[str, Any]] \| None]`	`None`
`mm_processor_kwargs` (InitVar) `config/model.py:436`	`InitVar[dict[str, Any] \| None]`	`None`
`mm_processor_cache_gb` (InitVar) `config/model.py:437`	`InitVar[float \| None]`	`None`
`mm_processor_cache_type` (InitVar) `config/model.py:438`	`InitVar[MMCacheType \| None]`	`None`
`mm_hasher_algorithm` (InitVar) `config/model.py:439`	`InitVar[MMHasherAlgorithm \| None]`	`None`
`mm_shm_cache_max_object_size_mb` (InitVar) `config/model.py:440`	`InitVar[int \| None]`	`None`
`mm_encoder_only` (InitVar) `config/model.py:441`	`InitVar[bool \| None]`	`None`
`mm_encoder_tp_mode` (InitVar) `config/model.py:442`	`InitVar[MMEncoderTPMode \| None]`	`None`
`mm_encoder_attn_backend` (InitVar) `config/model.py:443`	`InitVar[AttentionBackendEnum \| str \| None]`	`None`
`mm_encoder_attn_dtype` (InitVar) `config/model.py:444`	`InitVar[str \| None]`	`None`
`mm_encoder_fp8_scale_path` (InitVar) `config/model.py:445`	`InitVar[str \| None]`	`None`
`mm_encoder_fp8_scale_save_path` (InitVar) `config/model.py:446`	`InitVar[str \| None]`	`None`
`mm_encoder_fp8_scale_save_margin` (InitVar) `config/model.py:447`	`InitVar[float \| None]`	`None`
`interleave_mm_strings` (InitVar) `config/model.py:448`	`InitVar[bool \| None]`	`None`
`skip_mm_profiling` (InitVar) `config/model.py:449`	`InitVar[bool \| None]`	`None`
`video_pruning_rate` (InitVar) `config/model.py:450`	`InitVar[float \| None]`	`None`
`video_pruning_method` (InitVar) `config/model.py:451`	`InitVar[str \| None]`	`None`
`mm_tensor_ipc` (InitVar) `config/model.py:452`	`InitVar[MMTensorIPC]`	`None`
`mm_ipc_gpu_memory_gb` (InitVar) `config/model.py:453`	`InitVar[float \| None]`	`None`
`mm_device_do_normalize` (InitVar) `config/model.py:454`	`InitVar[bool \| None]`	`None`
`mm_processor_device` (InitVar) `config/model.py:455`	`InitVar[MMProcessorDevice \| None]`	`None`

MultiModalConfig#

config/multimodal.py:301, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`language_model_only` `config/multimodal.py:304`	`bool`	`False`	If True, disables all multimodal inputs by setting all modality limits to 0. Equivalent to setting `--limit-mm-per-prompt` to 0 for every modality.
`limit_per_prompt` `config/multimodal.py:307`	`MultiModalDummyOptions`	`Field(default_factory=MultiModalDummyOptions)`	The maximum number of input items and options allowed per prompt for each modality.
`enable_mm_embeds` `config/multimodal.py:326`	`bool`	`False`	If `True`, enables passing multimodal embeddings: for `LLM` class, this refers to tensor inputs under `multi_modal_data`; for the OpenAI-compatible server, this refers to chat messages with content `"type": "*_embeds"`.
`media_io_kwargs` `config/multimodal.py:339`	`dict[str, dict[str, Any]]`	`Field(default_factory=dict)`	Additional args passed to process media inputs, keyed by modalities. For example, to set num_frames for video, set `--media-io-kwargs '{"video": {"num_frames": 40} }'`
`mm_processor_kwargs` `config/multimodal.py:343`	`dict[str, object] \| None`	`None`	Arguments to be forwarded to the model's processor for multi-modal data, e.g., image processor. Overrides for the multi-modal processor obtained from `transformers.AutoProcessor.from_pretrained`.
`mm_device_do_normalize` `config/multimodal.py:352`	`bool \| None`	`True`	Move the do_normalize computation in the mm preprocessing to before the ViT, and let the device do it, so that CPU computation can be saved.
`mm_processor_cache_gb` `config/multimodal.py:357`	`float`	`Field(default=4, ge=0)`	The size (in GiB) of the multi-modal processor cache, which is used to avoid re-processing past multi-modal inputs.
`mm_processor_cache_type` `config/multimodal.py:369`	`MMCacheType`	`"lru"`	Type of cache to use for the multi-modal preprocessor/mapper. If `shm`, use shared memory FIFO cache. If `lru`, use mirrored LRU cache.
`mm_hasher_algorithm` `config/multimodal.py:372`	`MMHasherAlgorithm`	`"blake3"`	Hash algorithm to use for multi-modal input caching. Use `"sha256"` or `"sha512"` for FIPS-compliant deployments.
`mm_shm_cache_max_object_size_mb` `config/multimodal.py:375`	`int`	`Field(default=128, ge=0)`	Size limit (in MiB) for each object stored in the multi-modal processor shared memory cache. Only effective when `mm_processor_cache_type` is `"shm"`.
`mm_encoder_only` `config/multimodal.py:379`	`bool`	`False`	When enabled, skips the language component of the model.
`mm_encoder_tp_mode` `config/multimodal.py:385`	`MMEncoderTPMode`	`"weights"`	Indicates how to optimize multi-modal encoder inference using tensor parallelism (TP).
`mm_encoder_attn_backend` `config/multimodal.py:398`	`AttentionBackendEnum \| None`	`None`	Optional override for the multi-modal encoder attention backend when using vision transformers. Accepts any value from `vllm.v1.attention.backends.registry.AttentionBackendEnum` (e.g. `FLASH_ATTN`).
`mm_encoder_attn_dtype` `config/multimodal.py:402`	`Literal["fp8"] \| None`	`None`	Optional dtype override for ViT encoder attention. Set to `"fp8"` to enable FP8 quantization via the FlashInfer cuDNN backend. When set to `"fp8"` without a scale file, dynamic scaling is used automatically. See docs/features/quantization/fp8_vit_attn.md for details.
`mm_encoder_fp8_scale_path` `config/multimodal.py:407`	`str \| None`	`None`	Path to a JSON file containing per-layer FP8 Q/K/V scales for ViT encoder attention. When provided (with `mm_encoder_attn_dtype="fp8"`), static scaling is used. When omitted, dynamic scaling is used.
`mm_encoder_fp8_scale_save_path` `config/multimodal.py:411`	`str \| None`	`None`	When set with dynamic FP8 scaling (`mm_encoder_attn_dtype="fp8"` and no `mm_encoder_fp8_scale_path`), saves the calibrated scales to this file after the amax history buffer is full. The saved file can then be used as `mm_encoder_fp8_scale_path` in subsequent runs.
`mm_encoder_fp8_scale_save_margin` `config/multimodal.py:416`	`float`	`Field(default=1.5, gt=0.0)`	Safety margin multiplied onto scales when auto-saving. A value > 1 leaves headroom so that inputs with larger activations than the calibration set do not overflow FP8 range. Default 1.5.
`interleave_mm_strings` `config/multimodal.py:420`	`bool`	`False`	Enable fully interleaved support for multimodal prompts, while using --chat-template-content-format=string.
`skip_mm_profiling` `config/multimodal.py:423`	`bool`	`False`	When enabled, skips multimodal memory profiling and only profiles with language backbone model during engine initialization.
`video_pruning_rate` `config/multimodal.py:430`	`float \| None`	`Field(default=None, ge=0.0, lt=1.0)`	Fraction of video tokens to prune from each video. Value sits in range [0;1); pruning is enabled when it is greater than 0. The pruning algorithm is selected by `video_pruning_method`.
`video_pruning_method` `config/multimodal.py:435`	`VideoPruningMethod`	`"evs"`	Video token pruning algorithm applied when `video_pruning_rate` > 0: - "evs": Efficient Video Sampling. - "vidcom2": Video Compression Commander.
`mm_tensor_ipc` `config/multimodal.py:440`	`MMTensorIPC`	`"direct_rpc"`	IPC (inter-process communication) method for multimodal tensors. - "direct_rpc": Use msgspec serialization via RPC - "torch_shm": Use torch.multiprocessing shared memory for zero-copy IPC Defaults to "direct_rpc".
`allow_missing_mm_embeddings` `config/multimodal.py:445`	`bool`	`False`	Whether a pre-computed-embedding input may omit the `*_embeds` tensor.
`mm_ipc_gpu_memory_gb` `config/multimodal.py:459`	`float`	`Field(default=0, ge=0)`	Amount of GPU memory (in GiB) sequestered on the engine's device for GPU-side multimodal work in the API-server (frontend) process, such as hardware video decoding.

ObservabilityConfig#

config/observability.py:20, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`show_hidden_metrics_for_version` `config/observability.py:23`	`str \| None`	`None`	Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated metric has been hidden since the v0.7.0 release, you use `--show-hidden-metrics-for-version=0.7` as a temporary escape hatch while you migrate to new metrics. The metric is likely to be removed completely in an upcoming release.
`otlp_traces_endpoint` `config/observability.py:38`	`str \| None`	`None`	Target URL to which OpenTelemetry traces will be sent.
`collect_detailed_traces` `config/observability.py:41`	`list[DetailedTraceModules] \| None`	`None`	It makes sense to set this only if `--otlp-traces-endpoint` is set. If set, it will collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact.
`per_request_spec_decode_metrics` `config/observability.py:50`	`Literal["none", "summary", "detailed"]`	`"none"`	Include per-request speculative-decoding acceptance metrics in the response under `metrics.speculative_decoding`. `none` disables; `summary` adds mean acceptance length, draft acceptance rate, and the step-by-draft-length histogram; `detailed` additionally records the ordered per-step accepted/proposed arrays (one entry per verify step). Only reported for single-sequence requests (`n == 1`), …
`kv_cache_metrics` `config/observability.py:62`	`bool`	`False`	Enable KV cache residency metrics (lifetime, idle time, reuse gaps). Uses sampling to minimize overhead. Requires log stats to be enabled (i.e., --disable-log-stats not set).
`kv_cache_metrics_sample` `config/observability.py:67`	`float`	`Field(default=0.01, gt=0, le=1)`	Sampling rate for KV cache metrics (0.0, 1.0]. Default 0.01 = 1% of blocks.
`custom_histogram_buckets` `config/observability.py:70`	`dict[str, list[float]] \| None`	`None`	Custom Prometheus histogram bucket boundaries, as a JSON mapping from bucket-family key to a list of strictly increasing, positive, finite upper bounds. When a family key is present, its list replaces the default buckets for every histogram in that family; families not listed keep their defaults. Known families: `request_latency`, `time_to_first_token`, `inter_token_latency`, `iteration_tokens`, …
`cudagraph_metrics` `config/observability.py:82`	`bool`	`False`	Enable CUDA graph metrics (number of padded/unpadded tokens, runtime cudagraph dispatch modes, and their observed frequencies at every logging interval).
`enable_layerwise_nvtx_tracing` `config/observability.py:86`	`bool`	`False`	Enable layerwise NVTX tracing. This traces the execution of each layer or module in the model and attach information such as input/output shapes to nvtx range markers. Noted that this doesn't work with CUDA graphs enabled.
`enable_mfu_metrics` `config/observability.py:91`	`bool`	`False`	Enable Model FLOPs Utilization (MFU) metrics.
`enable_mm_processor_stats` `config/observability.py:94`	`bool`	`False`	Enable collection of timing statistics for multimodal processor operations. This is for internal use only (e.g., benchmarks) and is not exposed as a CLI argument.
`enable_logging_iteration_details` `config/observability.py:99`	`bool`	`False`	Enable detailed logging of iteration details. If set, vllm EngineCore will log iteration details This includes number of context/generation requests and tokens and the elapsed cpu time for the iteration.
`jit_monitor_mode` `config/observability.py:105`	`Literal["warn", "error"]`	`"warn"`	How to handle post-warmup JIT compilation events.
`jit_monitor_verbose` `config/observability.py:108`	`bool`	`False`	Log every monitored JIT compile with runtime details. This can emit many logs and add overhead, so it is intended for debugging.

UVAOffloadConfig#

config/offload.py:16, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`cpu_offload_gb` `config/offload.py:23`	`float`	`Field(default=0, ge=0)`	The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast …
`cpu_offload_params` `config/offload.py:34`	`set[str]`	`Field(default_factory=set)`	The set of parameter name segments to target for CPU offloading. Unmatched parameters are not offloaded. If this set is empty, parameters are offloaded non-selectively until the memory limit defined by `cpu_offload_gb` is reached. Examples: - For parameter name "mlp.experts.w2_weight": - "experts" or "experts.w2_weight" will match. - "expert" or "w2" will NOT match (must be exact segments). This …

PrefetchOffloadConfig#

config/offload.py:48, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`offload_group_size` `config/offload.py:54`	`int`	`Field(default=0, ge=0)`	Group every N layers together. Offload last `offload_num_in_group` layers of each group. Default is 0 (disabled). Example: group_size=8, num_in_group=2 offloads layers 6,7,14,15,22,23,... Unlike cpu_offload_gb, this uses explicit async prefetching to hide transfer latency.
`offload_num_in_group` `config/offload.py:62`	`int`	`Field(default=1, ge=1)`	Number of layers to offload per group. Must be <= offload_group_size. Default is 1.
`offload_prefetch_step` `config/offload.py:66`	`int`	`Field(default=1, ge=0)`	Number of layers to prefetch ahead. Higher values hide more latency but use more GPU memory. Default is 1.
`offload_params` `config/offload.py:70`	`set[str]`	`Field(default_factory=set)`	The set of parameter name segments to target for prefetch offloading. Unmatched parameters are not offloaded. If this set is empty, ALL parameters of each offloaded layer are offloaded. Uses segment matching: "w13_weight" matches "mlp.experts.w13_weight" but not "mlp.experts.w13_weight_scale".

OffloadConfig#

config/offload.py:80, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`offload_backend` `config/offload.py:83`	`OffloadBackend`	`"auto"`	The backend for weight offloading. Options: - "auto": Selects based on which sub-config has non-default values (prefetch if offload_group_size > 0, uva if cpu_offload_gb > 0). - "uva": UVA (Unified Virtual Addressing) zero-copy offloading. - "prefetch": Async prefetch with group-based layer offloading.
`uva` `config/offload.py:91`	`UVAOffloadConfig`	`Field(default_factory=UVAOffloadConfig)`	Parameters for UVA offloading backend.
`prefetch` `config/offload.py:94`	`PrefetchOffloadConfig`	`Field(default_factory=PrefetchOffloadConfig)`	Parameters for prefetch offloading backend.

EPLBConfig#

config/parallel.py:64, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`window_size` `config/parallel.py:67`	`int`	`Field(default=1000, gt=0)`	Window size for expert load recording.
`step_interval` `config/parallel.py:69`	`int`	`Field(default=3000, gt=0)`	Interval for rearranging experts in expert parallelism.
`num_redundant_experts` `config/parallel.py:77`	`int`	`Field(default=0, ge=0)`	Number of redundant experts to use for expert parallelism.
`log_balancedness` `config/parallel.py:80`	`bool`	`False`	Log the balancedness each step of expert parallelism. This is turned off by default since it will cause communication overhead.
`log_balancedness_interval` `config/parallel.py:85`	`int`	`Field(default=1, gt=0)`	Interval for logging the balancedness.
`use_async` `config/parallel.py:89`	`bool`	`True`	Whether to use non-blocking EPLB.
`policy` `config/parallel.py:94`	`EPLBPolicyOption`	`"default"`	The policy type for expert parallel load balancing (EPLB).
`communicator` `config/parallel.py:97`	`EPLBCommunicatorBackend \| None`	`None`	Backend for EPLB expert weight communication: - "torch_nccl": Use torch.distributed on the device process group - "torch_gloo": Use torch.distributed gloo with CPU staging - "torch_xccl": Use torch.distributed XCCL device P2P on XPU - "nixl": Use NIXL with staged send/recv buffers - "pynccl": Use PyNccl send/recv - None: Auto-select backend ("torch_xccl" on XPU, prefers "nixl" on CUDA, falls back …
`enable_migration_batching` `config/parallel.py:109`	`bool`	`False`	Schedule expert migrations in batches where each rank communicates with at most one peer. This reduces per-rank network contention at the cost of additional sequential communication steps. This option only applies to async EPLB and is disabled by default.

ParallelConfig#

config/parallel.py:137, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`pipeline_parallel_size` `config/parallel.py:140`	`int`	`Field(default=1, ge=1)`	Number of pipeline parallel groups.
`tensor_parallel_size` `config/parallel.py:142`	`int`	`Field(default=1, ge=1)`	Number of tensor parallel groups.
`prefill_context_parallel_size` `config/parallel.py:144`	`int`	`Field(default=1, ge=1)`	Number of ranks that split prefill sequence computation. PCP expands the process world size but does not increase the KV-cache shard count.
`data_parallel_size` `config/parallel.py:147`	`int`	`Field(default=1, ge=1)`	Number of data parallel groups. MoE layers will be sharded according to the product of the tensor, prefill-context, and data parallel sizes.
`data_parallel_size_local` `config/parallel.py:150`	`int`	`Field(default=1, ge=0)`	Number of local data parallel groups. A value of 0 is a sentinel used by the engine-args layer to signal that data parallelism was specified externally (see `ParallelConfig.__post_init__`).
`data_parallel_rank` `config/parallel.py:154`	`int`	`Field(default=0, ge=0)`	Rank of the data parallel group. The runtime check at ``__post_init__`` further bounds this by ``data_parallel_size``.
`data_parallel_rank_local` `config/parallel.py:157`	`int \| None`	`None`	Local rank of the data parallel group, set only in SPMD mode.
`data_parallel_master_ip` `config/parallel.py:159`	`str`	`"127.0.0.1"`	IP of the data parallel master.
`data_parallel_rpc_port` `config/parallel.py:161`	`int`	`Field(default=29550, ge=1, le=65535)`	Fixed port for data parallel messaging, shared by all nodes.
`dp_sync_interval` `config/parallel.py:163`	`int`	`Field(default=16, ge=1)`	Steps between DP finish-sync all-reduces; must match across DP ranks.
`data_parallel_master_port` `config/parallel.py:165`	`int`	`29500`	Port of the data parallel master.
`data_parallel_backend` `config/parallel.py:167`	`DataParallelBackend`	`"mp"`	Backend to use for data parallel, either "mp" or "ray".
`data_parallel_external_lb` `config/parallel.py:169`	`bool`	`False`	Whether to use "external" DP LB mode. Applies only to online serving and when data_parallel_size > 0. This is useful for a "one-pod-per-rank" wide-EP setup in Kubernetes. Supported only for MoE deployments; non-MoE models should use independent vLLM instances without --data-parallel-* arguments. Set implicitly when --data-parallel-rank is provided explicitly to vllm serve.
`data_parallel_hybrid_lb` `config/parallel.py:176`	`bool`	`False`	Whether to use "hybrid" DP LB mode. Applies only to online serving and when data_parallel_size > 0. Enables running an AsyncLLM and API server on a "per-node" basis where vLLM load balances between local data parallel ranks, but an external LB balances between vLLM nodes/replicas. Set explicitly in conjunction with --data-parallel-start-rank.
`is_moe_model` `config/parallel.py:183`	`bool \| None`	`None`	Whether the deployed model is MoE (if known).
`enable_hc_sp` `config/parallel.py:185`	`bool`	`False`	Enable sequence parallelism for HyperConnection (HC) computation. Currently supported by Qwen4Exp on CUDA/ROCm with TP>1 and PP=1 (ignored when TP=1). Does not change MoE parallelism. MoE SP may enable HC SP independently of this flag.
`enable_expert_parallel` `config/parallel.py:190`	`bool`	`False`	Use expert parallelism instead of tensor parallelism for MoE layers.
`enable_batch_sharded_sampling` `config/parallel.py:192`	`bool \| None`	`None`	Use sharded sampling across tensor parallel ranks. Each rank samples a slice of the batch instead of every rank sampling all of it. Currently defaults to False if not set. Enabling it explicitly raises when the config cannot support it (`tensor_parallel_size` must be > 1, `max_num_seqs` at least `tensor_parallel_size`, and `max_logprobs` non-negative). Models opt in by implementing …
`enable_ep_weight_filter` `config/parallel.py:199`	`bool`	`False`	Skip non-local expert weights during model loading when expert parallelism is active. Each rank only reads its own expert shard from disk, which can drastically reduce storage I/O for MoE models with per-expert weight tensors (e.g. DeepSeek, Mixtral, Kimi-K2.5). Has no effect on 3D fused-expert checkpoints (e.g. GPT-OSS) or non-MoE models.
`enable_eplb` `config/parallel.py:206`	`bool`	`False`	Enable expert parallelism load balancing for MoE layers.
`eplb_config` `config/parallel.py:208`	`EPLBConfig`	`Field(default_factory=EPLBConfig)`	Expert parallelism configuration.
`expert_placement_strategy` `config/parallel.py:210`	`ExpertPlacementStrategy`	`"linear"`	The expert placement strategy for MoE layers:
`all2all_backend` `config/parallel.py:220`	`All2AllBackend`	`"allgather_reducescatter"`	All2All backend for MoE expert parallel communication. Available options:
`max_parallel_loading_workers` `config/parallel.py:236`	`int \| None`	`Field(default=None, ge=1)`	Maximum number of parallel loading workers when loading model sequentially in multiple batches. To avoid RAM OOM when using tensor parallel and large models.
`disable_custom_all_reduce` `config/parallel.py:241`	`bool`	`False`	Disable the custom all-reduce kernel and fall back to NCCL.
`enable_shm_tensor_arena` `config/parallel.py:244`	`bool`	`False`	Route large CPU tensors (e.g. multimodal ``pixel_values``) in the engine→worker broadcast through a zero-copy shared-memory arena instead of sending a copy to every reader. Experimental: may be reconciled with vLLM's existing multimodal shm tensor caching in a future release. Opt-in; most beneficial for multimodal serving with large images and tensor parallelism. Active only when all queue …
`enable_elastic_ep` `config/parallel.py:253`	`bool`	`False`	Enable elastic expert parallelism with stateless NCCL groups for DP/EP.
`elastic_ep_max_dp_size` `config/parallel.py:255`	`int`	`Field(default=None, ge=1)`	Maximum data parallel size supported by elastic expert parallelism.
`enable_dbo` `config/parallel.py:258`	`bool`	`False`	Enable dual batch overlap for the model executor.
`ubatch_size` `config/parallel.py:260`	`int`	`Field(default=0, ge=0)`	Number of ubatch size.
`dbo_decode_token_threshold` `config/parallel.py:263`	`int`	`Field(default=32, ge=0)`	The threshold for dual batch overlap for batches only containing decodes. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch.
`dbo_prefill_token_threshold` `config/parallel.py:268`	`int`	`Field(default=512, ge=0)`	The threshold for dual batch overlap for batches that contain one or more prefills. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch.
`disable_nccl_for_dp_synchronization` `config/parallel.py:274`	`bool \| None`	`None`	Forces the dp synchronization logic in vllm/v1/worker/dp_utils.py to use Gloo instead of NCCL for its all reduce.
`ray_workers_use_nsight` `config/parallel.py:281`	`bool`	`False`	Whether to profile Ray workers with nsight, see https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler.
`ray_runtime_env` `config/parallel.py:284`	`RuntimeEnv \| None`	`None`	Ray runtime environment to pass to distributed workers.
`placement_group` `config/parallel.py:287`	`PlacementGroup \| None`	`None`	ray distributed model workers placement group.
`distributed_executor_backend` `config/parallel.py:290`	`str \| DistributedExecutorBackend \| type[Executor] \| None`	`None`	Backend to use for distributed model workers, either "ray" or "mp" (multiprocessing). If the product of pipeline_parallel_size and tensor_parallel_size is less than or equal to the number of GPUs available, "mp" will be used to keep processing on a single host. Otherwise, an error will be raised. To use "mp" you must also set nnodes, and to use "ray" you must manually set …
`worker_cls` `config/parallel.py:306`	`str`	`"auto"`	The full name of the worker class to use. If "auto", the worker class will be determined based on the platform.
`worker_extension_cls` `config/parallel.py:309`	`str`	`""`	The full name of the worker extension class to use. The worker extension class is dynamically inherited by the worker class. This is used to inject new attributes and methods to the worker class for use in collective_rpc calls. It may also define `init_worker_extension(self)`, called after device and distributed initialization but before model loading, and `destroy_worker_extension(self)`, called …
`master_addr` `config/parallel.py:317`	`str`	`"127.0.0.1"`	distributed master address for multi-node distributed inference when distributed_executor_backend is mp.
`master_port` `config/parallel.py:320`	`int`	`29501`	distributed master port for multi-node distributed inference when distributed_executor_backend is mp.
`node_rank` `config/parallel.py:323`	`int`	`Field(default=0, ge=0)`	distributed node rank for multi-node distributed inference when distributed_executor_backend is mp.
`nnodes` `config/parallel.py:326`	`int`	`Field(default=1, ge=1)`	num of nodes for multi-node distributed inference when distributed_executor_backend is mp.
`numa_bind` `config/parallel.py:329`	`bool`	`False`	Enable NUMA binding for GPU worker subprocesses.
`numa_bind_nodes` `config/parallel.py:336`	`list[int] \| None`	`None`	NUMA node to bind each GPU worker to.
`numa_bind_cpus` `config/parallel.py:345`	`list[str] \| None`	`None`	Optional CPU lists to bind each GPU worker to.
`assigned_physical_gpu_ids` `config/parallel.py:355`	`list[int] \| None`	`None`	Mapping from vLLM-local logical GPU IDs to physical GPU IDs.
`distributed_timeout_seconds` `config/parallel.py:364`	`int \| None`	`None`	Timeout in seconds for distributed operations (e.g., init_process_group). If set, this value is passed to torch.distributed.init_process_group as the timeout parameter. If None, PyTorch's default timeout is used (600s for NCCL). Increase this for multi-node setups where model downloads may be slow.
`cpu_distributed_timeout_seconds` `config/parallel.py:370`	`int \| None`	`None`	Timeout (in seconds) for cpu communication groups. If None, PyTorch's default timeout is used (1800s for gloo).
`world_size` `config/parallel.py:374`	`int`	`Field(init=False)`	world_size is TPxPP, it affects the number of workers we create.
`rank` `config/parallel.py:377`	`int`	`0`	Global rank in distributed setup.
`_data_parallel_master_port_list` `config/parallel.py:380`	`list[int]`	`Field(default_factory=list)`	List of open port auto-queried for data parallel messaging. Set to be private as it's not intended to be configured by users.
`_coord_store_port` `config/parallel.py:385`	`int`	`0`	Port of the coordination TCPStore. Can be set by the API server; workers connect as clients to exchange self-picked group ports at runtime.
`decode_context_parallel_size` `config/parallel.py:389`	`int`	`Field(default=1, ge=1)`	Number of ranks that shard the decode KV cache. DCP does not expand the process world size. Without PCP, DCP reuses TP ranks. With PCP, DCP either spans the PCP axis or the full TP x PCP block.
`dcp_kv_cache_interleave_size` `config/parallel.py:394`	`int`	`1`	Interleave size of kv_cache storage while using DCP. dcp_kv_cache_interleave_size has been replaced by cp_kv_cache_interleave_size, and will be deprecated when PCP is fully supported.
`dcp_comm_backend` `config/parallel.py:401`	`DCPCommBackend \| None`	`None`	Communication backend for Decode Context Parallel (DCP). - "ag_rs": AllGather + ReduceScatter (existing behavior) - "a2a": All-to-All exchange of partial outputs + LSE, then combine with Triton kernel. Reduces NCCL calls from 3 to 2 per layer for MLA models.
`dcp_q_replicate` `config/parallel.py:412`	`bool \| None`	`None`	Replicate the MLA query projection within each DCP group so decode can skip the query all-gather.
`cp_kv_cache_interleave_size` `config/parallel.py:423`	`int`	`1`	Interleave size of kv_cache storage while using DCP. Store interleave_size tokens on dcp_rank i, then store next interleave_size tokens on dcp_rank i+1. Interleave_size=1: token-level alignment, where token `i` is stored on dcp_rank `i % dcp_world_size`. Interleave_size=block_size: block-level alignment, where tokens are first populated to the preceding ranks. Tokens are then stored in (rank i+1, …
`_allow_auto_resolve_cp_interleave_size` `config/parallel.py:440`	`bool`	`True`	Whether NIXL may select the interleave size automatically.
`data_parallel_index` `config/parallel.py:443`	`int`	`Field(init=False)`	Equal to the data parallel rank but not used for torch process groups and not overridden for dense models.
`_api_process_count` `config/parallel.py:447`	`int`	`Field(default=1, gt=0)`	The number of API processes initialized.
`_api_process_rank` `config/parallel.py:456`	`int`	`Field(default=0, ge=-1)`	The rank of this API process, or `-1` for engine core processes under API server scale-out.
`enable_fault_tolerance` `config/parallel.py:466`	`bool`	`False`	Enable fault tolerance for detailed error recovery, such as scaling down fault DPEngineCore.
`fault_tolerance_config` `config/parallel.py:471`	`FaultToleranceConfig`	`Field(default_factory=FaultToleranceConfig)`	The configurations for fault tolerance.

PoolerConfig#

config/pooler.py:30, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`task` `config/pooler.py:33`	`PoolingTask \| None`	`None`	The task used for pooling.
`pooling_type` `config/pooler.py:38`	`SequencePoolingType \| TokenPoolingType \| None`	`None`	The pooling method used for pooling.
`seq_pooling_type` `config/pooler.py:50`	`SequencePoolingType \| None`	`None`	The pooling method used for sequence pooling.
`tok_pooling_type` `config/pooler.py:55`	`TokenPoolingType \| None`	`None`	The pooling method used for tokenwise pooling.
`use_activation` `config/pooler.py:60`	`bool \| None`	`None`	Whether to apply activation function to the pooler outputs. `None` uses the pooler's default, which is `True` in most cases.
`enable_flash_late_interaction` `config/pooler.py:66`	`bool`	`True`	Whether the engine-side late-interaction scorer may use the fused flash-maxsim Triton kernel. Disabled automatically when the API server is started with `--disable-flash-late-interaction`; the reference scorer is used instead.
`dimensions` `config/pooler.py:75`	`int \| None`	`None`	Reduce the dimensions of embeddings if model support matryoshka representation. Defaults to None.
`enable_chunked_processing` `config/pooler.py:80`	`bool`	`False`	Whether to enable chunked processing for long inputs that exceed the model's maximum position embeddings. When enabled, long inputs will be split into chunks, processed separately, and then aggregated using weighted averaging. This allows embedding models to handle arbitrarily long text without CUDA errors. Multimodal inputs are not supported. Defaults to False.
`max_embed_len` `config/pooler.py:88`	`int \| None`	`None`	Maximum input length allowed for embedding generation. When set, allows inputs longer than max_embed_len to be accepted for embedding models. When an input exceeds max_embed_len, it will be handled according to the original max_model_len validation logic. Defaults to None (i.e. set to max_model_len).
`logit_mean` `config/pooler.py:98`	`float \| None`	`None`	If provided, subtract this value from classification logits before activation. Used for affine score calibration (Platt scaling): activation((logit - logit_mean) / logit_sigma). Defaults to None.
`logit_sigma` `config/pooler.py:105`	`float \| None`	`None`	If provided, divide the classification logits by this value after mean subtraction. Used for affine score calibration (Platt scaling): activation((logit - logit_mean) / logit_sigma). Defaults to None.
`step_tag_id` `config/pooler.py:113`	`int \| None`	`None`	If set, only the score corresponding to the `step_tag_id` in the generated sentence should be returned. Otherwise, the scores for all tokens are returned.
`returned_token_ids` `config/pooler.py:119`	`list[int] \| None`	`None`	A list of indices for the vocabulary dimensions to be extracted, such as the token IDs of `good_token` and `bad_token` in the `math-shepherd-mistral-7b-prm` model.

ProfilerConfig#

config/profiler.py:67, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`profiler` `config/profiler.py:70`	`ProfilerKind \| None`	`None`	Which profiler to use. Defaults to None. Options are:
`torch_profiler_dir` `config/profiler.py:77`	`str`	`""`	Directory to save torch profiler traces. Both AsyncLLM's CPU traces and worker's traces (CPU & GPU) will be saved under this directory. Note that it must be an absolute path.
`torch_profiler_activities` `config/profiler.py:82`	`list[TorchProfilerActivity] \| None`	`Field(default=None, min_length=1)`	Activities recorded by workers using the torch profiler. When unset, each worker uses its platform default: CPU; CPU and CUDA; or CPU and XPU.
`proton_profiler_dir` `config/profiler.py:88`	`str`	`""`	Directory to save Triton Proton profiles. Each worker writes a separate rank-qualified file.
`proton_context` `config/profiler.py:92`	`ProtonContext`	`"shadow"`	Proton context source. ``shadow`` records explicit scopes with low overhead; ``python`` records Python call stacks.
`proton_data` `config/profiler.py:96`	`ProtonData`	`"tree"`	Proton output type. ``tree`` produces Hatchet data and ``trace`` produces a Chrome trace.
`proton_backend` `config/profiler.py:100`	`ProtonBackend \| None`	`None`	Proton GPU backend. ``None`` lets Proton select CUPTI automatically.
`proton_mode` `config/profiler.py:103`	`str \| None`	`None`	Optional backend-specific Proton mode string, such as ``pcsampling``.
`proton_hook` `config/profiler.py:106`	`ProtonHook \| None`	`None`	Optional Proton hook. Use ``triton`` to add Triton launch metadata.
`proton_output_format` `config/profiler.py:109`	`ProtonOutputFormat \| None`	`None`	Optional format passed to Proton when finalizing a profile. ``None`` uses the default format for ``proton_data``.
`proton_graph_attribution` `config/profiler.py:113`	`bool`	`False`	Observe CUDA graph capture so replayed kernels can be attributed. Requires Triton >= 3.7 and ``proton_data='tree'``.
`torch_profiler_with_stack` `config/profiler.py:117`	`bool`	`True`	If `True`, enables stack tracing in the torch profiler. Enabled by default as it is useful for debugging. Can be disabled via --profiler-config.torch_profiler_with_stack=false CLI flag.
`torch_profiler_with_flops` `config/profiler.py:122`	`bool`	`False`	If `True`, enables FLOPS counting in the torch profiler. Disabled by default.
`torch_profiler_use_gzip` `config/profiler.py:125`	`bool`	`True`	If `True`, saves torch profiler traces in gzip format. Enabled by default
`torch_profiler_dump_cuda_time_total` `config/profiler.py:128`	`bool`	`True`	If `True`, dumps total CUDA time in torch profiler traces. Enabled by default.
`torch_profiler_record_shapes` `config/profiler.py:131`	`bool`	`False`	If `True`, records tensor shapes in the torch profiler. Disabled by default.
`torch_profiler_with_memory` `config/profiler.py:134`	`bool`	`False`	If `True`, enables memory profiling in the torch profiler. Disabled by default.
`capture_torch_profiler` `config/profiler.py:138`	`bool`	`False`	If `True`, enables a torch profiler during CUDA graph capture on rank 0. Traces are saved to a `capture_traces` subdirectory under `torch_profiler_dir`. Requires `profiler` to be set to 'torch'.
`detailed_trace_annotation` `config/profiler.py:143`	`bool`	`False`	If `True`, uses detailed annotations with roofline metrics (sk, sqsq, sqsk) in profiler trace events. If `False`, uses simple annotations with only context/generation request counts and token counts. Disabled by default.
`ignore_frontend` `config/profiler.py:149`	`bool`	`False`	If `True`, disables the front-end profiling of AsyncLLM when using the 'torch' profiler. This is needed to reduce overhead when using delay/limit options, since the front-end profiling does not track iterations and will capture the entire range.
`delay_iterations` `config/profiler.py:156`	`int`	`Field(default=0, ge=0)`	Number of engine iterations to skip before starting profiling. Defaults to 0, meaning profiling starts immediately after receiving /start_profile.
`max_iterations` `config/profiler.py:161`	`int`	`Field(default=0, ge=0)`	Maximum number of engine iterations to profile after starting profiling. Defaults to 0, meaning no limit.
`warmup_iterations` `config/profiler.py:176`	`int`	`Field(default=0, ge=0)`	Number of warmup iterations for PyTorch profiler schedule. During warmup, the profiler runs but data is discarded. This helps reduce noise from JIT compilation and other one-time costs in the profiled trace. Defaults to 0 (schedule-based profiling disabled, recording all iterations). Set to a positive value (e.g., 2) to enable schedule-based profiling.
`active_iterations` `config/profiler.py:184`	`int`	`Field(default=5, ge=1)`	Number of active iterations for PyTorch profiler schedule. This is the number of iterations where profiling data is actually collected. Defaults to 5 active iterations.
`wait_iterations` `config/profiler.py:190`	`int`	`Field(default=0, ge=0)`	Number of wait iterations for PyTorch profiler schedule. During wait, the profiler is completely off with zero overhead. This allows skipping initial iterations before warmup begins. Defaults to 0 (no wait period).

QuantSpec#

config/quantization.py:78, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`weight` `config/quantization.py:85`	`QuantKeyField`	`None`	Weight quantization key, or a name from QUANT_KEY_NAMES.
`activation` `config/quantization.py:88`	`QuantKeyField`	`None`	Activation quantization key, or a name from QUANT_KEY_NAMES.

QuantizationConfigArgs#

config/quantization.py:108, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`linear` `config/quantization.py:115`	`QuantSpec \| None`	`None`	Spec applied to ``LinearBase`` layers.
`moe` `config/quantization.py:118`	`QuantSpec \| None`	`None`	Spec applied to ``FusedMoEFactory`` layers.
`ignore` `config/quantization.py:121`	`list[str]`	`Field(default_factory=list)`	Layers to skip quantization for. Online quantization also supports fnmatch-style patterns.
`targets` `config/quantization.py:125`	`dict[str, str] \| None`	`None`	Per-layer online quantization overrides, keyed by exact layer name or regex patterns with a `re:`, or fnmatch-style patterns for online quantization, mapping to an online shorthand name (see `_ONLINE_SHORTHANDS`). A layer that matches no pattern is left unquantized. Mutually exclusive with `linear` and `moe`.

ReasoningConfig#

config/reasoning.py:13, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`reasoning_parser` `config/reasoning.py:22`	`str`	`""`	The name of the ReasoningParser to use for this model.
`reasoning_start_str` `config/reasoning.py:24`	`str`	`""`	String that indicates the start of reasoning.
`reasoning_end_str` `config/reasoning.py:26`	`str`	`""`	String forced when the thinking budget is exhausted.
`_reasoning_start_token_ids` `config/reasoning.py:29`	`list[int] \| None`	`field(default=None, init=False, repr=False)`	Private backing field for `reasoning_start_token_ids`. Set by `initialize_token_ids`. Not intended to be configured directly.
`_reasoning_end_token_ids` `config/reasoning.py:34`	`list[int] \| None`	`field(default=None, init=False, repr=False)`	Private backing field for forced reasoning end token IDs.
`_natural_reasoning_end_token_ids` `config/reasoning.py:38`	`list[int] \| None`	`field(default=None, init=False, repr=False)`	Token IDs that naturally terminate reasoning, as defined by the parser.
`_enabled` `config/reasoning.py:43`	`bool`	`field(default=False, init=False, repr=False)`	Private field indicating whether reasoning token IDs have been initialized. Set to True by `initialize_token_ids` once token IDs are initialized.

SchedulerConfig#

config/scheduler.py:25, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`max_model_len` (InitVar) `config/scheduler.py:28`	`InitVar[int]`	required	Maximum length of a sequence (including prompt and generated text).
`is_encoder_decoder` (InitVar) `config/scheduler.py:34`	`InitVar[bool]`	required	True if the model is an encoder-decoder model.
`DEFAULT_MAX_NUM_BATCHED_TOKENS` (ClassVar) `config/scheduler.py:41`	`ClassVar[int]`	`2048`
`DEFAULT_MAX_NUM_BATCHED_TOKENS_FOR_BATCHED_DP` (ClassVar) `config/scheduler.py:42`	`ClassVar[int]`	`256`
`DEFAULT_MAX_NUM_SEQS` (ClassVar) `config/scheduler.py:43`	`ClassVar[int]`	`128`
`runner_type` `config/scheduler.py:45`	`RunnerType`	`"generate"`	The runner type to launch for the model.
`max_num_batched_tokens` `config/scheduler.py:48`	`int`	`Field(default=DEFAULT_MAX_NUM_BATCHED_TOKENS, ge=1)`	Maximum number of tokens that can be processed in a single iteration.
`max_num_scheduled_tokens` `config/scheduler.py:55`	`int \| None`	`Field(default=None, ge=1)`	Maximum number of tokens that the scheduler may issue in a single iteration.
`max_num_seqs` `config/scheduler.py:62`	`int`	`Field(default=DEFAULT_MAX_NUM_SEQS, ge=1)`	Maximum number of sequences to be processed in a single iteration.
`max_num_active_seqs` `config/scheduler.py:69`	`int \| None`	`Field(default=None, ge=1)`	Maximum number of requests the scheduler admits into RUNNING.
`long_prefill_token_threshold` `config/scheduler.py:79`	`int`	`Field(default=0, ge=0)`	For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. 0 disables the cap (default).
`long_prefill_token_threshold_adaptive` `config/scheduler.py:86`	`bool`	`Field(default=False)`	Floor the effective long prefill token threshold at a fair share of the token budget: max_num_batched_tokens divided by the number of queued and running requests. Only applies when long_prefill_token_threshold is nonzero.
`max_num_queued_reqs` `config/scheduler.py:92`	`int \| None`	`Field(default=None, ge=0)`	Maximum number of requests that can be in-flight (waiting or running) at the same time, or None for no limit. When the limit is reached, new requests are rejected with HTTP 503 so the client can retry on another instance. This bounds vLLM's otherwise unbounded request queue and is primarily a coarse capacity valve.
`max_num_queued_tokens` `config/scheduler.py:105`	`int \| None`	`Field(default=None, ge=0)`	Maximum total prompt tokens of requests currently in the prefill phase, or None for no limit. When the limit is reached, new requests are rejected with HTTP 503.
`enable_chunked_prefill` `config/scheduler.py:134`	`bool`	`True`	If True, prefill requests can be chunked based on the remaining `max_num_batched_tokens`.
`is_multimodal_model` `config/scheduler.py:142`	`bool`	`False`	True if the model is multimodal.
`max_num_encoder_input_tokens` `config/scheduler.py:146`	`int`	`Field(init=False)`	Multimodal encoder compute budget, only used in V1.
`encoder_cache_size` `config/scheduler.py:153`	`int`	`Field(init=False)`	Multimodal encoder cache size, only used in V1.
`policy` `config/scheduler.py:159`	`SchedulerPolicy`	`"fcfs"`	The scheduling policy to use:
`disable_chunked_mm_input` `config/scheduler.py:167`	`bool`	`False`	If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. Only used in V1 This ensures that if a request has a mixed prompt (like text tokens TTTT followed by image tokens IIIIIIIIII) where only some image tokens can be scheduled (like TTTTIIIII, leaving IIIII), it will be scheduled as TTTT in one step and IIIIIIIIII in the next.
`scheduler_cls` `config/scheduler.py:177`	`str \| type[object] \| None`	`None`	The scheduler class to use. "vllm.v1.core.sched.scheduler.Scheduler" is the default scheduler. Can be a class directly or the path to a class of form "mod.custom_class".
`disable_hybrid_kv_cache_manager` `config/scheduler.py:182`	`bool \| None`	`None`	If set to True, KV cache manager will allocate the same size of KV cache for all attention layers even if there are multiple type of attention layers like full attention and sliding window attention. If set to None, the default value will be determined based on the environment and starting configuration.
`scheduler_reserve_full_isl` `config/scheduler.py:190`	`bool`	`True`	If True, the scheduler checks whether the full input sequence length fits in the KV cache before admitting a new request, rather than only checking the first chunk. Prevents over-admission and KV cache thrashing with chunked prefill.
`watermark` `config/scheduler.py:196`	`float`	`Field(default=0.0, ge=0.0, lt=1.0)`	Fraction of total KV cache blocks to keep free (the watermark) when admitting waiting or preempted requests into the running queue. This headroom helps avoid frequent KV cache eviction and the resulting repeated preemption of requests when GPU memory is scarce. Must be in the range [0.0, 1.0); 0.0 (the default) disables the watermark.
`prefill_schedule_interval` `config/scheduler.py:203`	`int`	`Field(default=1, ge=1)`	Only admit prefill once every N engine steps, leaving the steps in between to decode alone. 1 (the default) admits prefill on every step.
`async_scheduling` `config/scheduler.py:211`	`bool \| None`	`None`	If set to False, disable async scheduling. Async scheduling helps to avoid gaps in GPU utilization, leading to better latency and throughput.
`stream_interval` `config/scheduler.py:216`	`int`	`Field(default=1, ge=1)`	The interval (or buffer size) for streaming in terms of token length. A smaller value (1) makes streaming smoother by sending each token immediately, while a larger value (e.g., 10) reduces host overhead and may increase throughput by batching multiple tokens before sending.

SpeculativeConfig#

config/speculative.py:387, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`enforce_eager` `config/speculative.py:390`	`bool \| None`	`None`	Override the default enforce_eager from model_config
`num_speculative_tokens` `config/speculative.py:393`	`int`	`Field(default=None, gt=0)`	The number of speculative tokens, if provided. It will default to the number in the draft model config if present, otherwise, it is required.
`model` `config/speculative.py:396`	`str \| None`	`None`	The name of the draft model, eagle head, or additional weights, if provided.
`method` `config/speculative.py:399`	`SpeculativeMethod \| None`	`None`	The name of the speculative method to use. If users provide and set the `model` param, the speculative method type will be detected automatically if possible, if `model` param is not provided, the method name must be provided.
`draft_tensor_parallel_size` `config/speculative.py:407`	`int \| None`	`Field(default=None, ge=1)`	The degree of the tensor parallelism for the draft model. Can only be 1 or the same as the target model's tensor parallel size.
`tensor_parallel_size` `config/speculative.py:410`	`int \| None`	`None`	Users should pass "draft_tensor_parallel_size". This parameter's purpose is to warn users when they mistakenly provide the wrong argument.
`quantization` `config/speculative.py:415`	`me_quant.QuantizationMethods \| str \| None`	`None`	Quantization method that was used to quantize the draft model weights. If `None`, we assume the model weights are not quantized. Note that it only takes effect when using the draft model-based speculative method.
`moe_backend` `config/speculative.py:419`	`MoEBackend \| None`	`None`	MoE backend to use for the draft model. When `None`, the draft model inherits the target model's `--moe-backend` setting. Useful when the drafter and generator require different MoE kernels (e.g. quantized generator with unquantized drafter).
`attention_backend` `config/speculative.py:424`	`AttentionBackendEnum \| None`	`None`	Attention backend to use for the draft model. When `None`, the backend is automatically selected. Useful when the drafter requires a different attention backend (e.g. DFlash needs a backend that supports non-causal attention).
`kv_cache_dtype` `config/speculative.py:428`	`CacheDType \| str \| None`	`None`	KV cache dtype for the draft model. When `None`, the draft inherits the target model's `--kv-cache-dtype`. Custom dtypes registered by platform backends are accepted.
`max_model_len` `config/speculative.py:432`	`int \| None`	`Field(default=None, ge=1)`	The maximum model length of the draft model. Used when testing the ability to skip speculation for some sequences.
`revision` `config/speculative.py:435`	`str \| None`	`None`	The specific model version to use for the draft model. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.
`code_revision` `config/speculative.py:439`	`str \| None`	`None`	The specific revision to use for the draft model code on Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.
`index_share_for_mtp_iteration` `config/speculative.py:443`	`bool \| None`	`None`	Override whether MTP iterations reuse the first step's sparse indices. If `None`, use the value from the draft model's Hugging Face config.
`disable_padded_drafter_batch` `config/speculative.py:448`	`bool`	`False`	Disable input padding for speculative decoding. If set to True, speculative input batches can contain sequences of different lengths, which may only be supported by certain attention backends. This currently only affects the EAGLE method of speculation.
`disable_eagle_block_drop` `config/speculative.py:453`	`bool`	`False`	Disable dropping the trailing prefix-cache block for EAGLE-like speculative methods. This is an experimental option for measuring the acceptance-rate impact of reusing that block. It does not disable the speculative drafter itself.
`draft_lm_head_quantization` `config/speculative.py:458`	`Literal["fp8", "nvfp4"] \| None`	`None`	Quantize the drafter's copy of a target lm_head it shares (EAGLE and MTP). "fp8" uses per-row weight and per-token activation scales on the platform's FP8 linear kernel; "nvfp4" is weight-only on the platform's W4A16 kernel. The target verifies with its own head, so only the acceptance length can change. Model Runner V2 only.
`use_local_argmax_reduction` `config/speculative.py:464`	`bool`	`False`	Use vocab-parallel local argmax instead of all-gathering full logits for draft token generation. Reduces communication from O(vocab_size) to O(2 * tp_size) per token. Only applies to greedy draft selection in non-tree speculation.
`use_heterogeneous_vocab` `config/speculative.py:470`	`bool`	`False`	Allow draft and target models to use different vocabularies. When enabled, builds a token-level intersection at init and constrains draft logits to shared tokens only (TLI algorithm). Requires method='draft_model'.
`prompt_lookup_max` `config/speculative.py:477`	`int \| None`	`Field(default=None, ge=1)`	Maximum size of ngram token window when using Ngram proposer, required when method is set to ngram.
`prompt_lookup_min` `config/speculative.py:480`	`int \| None`	`Field(default=None, ge=1)`	Minimum size of ngram token window when using Ngram proposer, if provided. Defaults to 1.
`parallel_drafting` `config/speculative.py:485`	`bool`	`False`	Enable parallel drafting, where all speculative tokens are generated in parallel rather than sequentially. This can improve performance but requires the speculative model be trained to support parallel drafting. Only compatible with EAGLE and draft model methods.
`target_model_config` `config/speculative.py:492`	`SkipValidation[ModelConfig]`	`None`	The configuration of the target model.
`target_parallel_config` `config/speculative.py:494`	`SkipValidation[ParallelConfig]`	`None`	The parallel configuration for the target model.
`num_speculative_tokens_per_batch_size` `config/speculative.py:498`	`list[tuple[int, int, int]] \| None`	`None`	Batch-size schedule used to dynamically choose speculative-token count.
`draft_model_config` `config/speculative.py:506`	`SkipValidation[ModelConfig]`	`None`	The configuration of the draft model initialized internal.
`draft_parallel_config` `config/speculative.py:508`	`SkipValidation[ParallelConfig]`	`None`	The parallel configuration for the draft model initialized internal.
`suffix_decoding_max_tree_depth` `config/speculative.py:512`	`int`	`24`	The maximum depth of the suffix decoding global and prompt trees. The tree depth limits the sum of the prefix match and speculation lengths.
`suffix_decoding_max_cached_requests` `config/speculative.py:516`	`int`	`10000`	The maximum number of requests to cache in the global suffix tree. If exceeded, will trigger eviction in FIFO order. If set to 0, the global suffix tree is disabled and past responses are not cached (prompt trees are still used).
`suffix_decoding_max_spec_factor` `config/speculative.py:522`	`float`	`1.0`	The maximum spec factor for suffix decoding. The spec factor controls speculation lengths based on the prefix match length: max_spec_tokens = max_spec_factor * prefix_match_length.
`suffix_decoding_min_token_prob` `config/speculative.py:527`	`float`	`0.1`	The minimum token probability for suffix decoding. Will only speculate tokens with estimated probability (based on frequency counts) greater than or equal to this value.
`draft_load_config` `config/speculative.py:532`	`LoadConfig \| None`	`None`	Load config for the draft model. If not specified, will use the load config from the target model.
`rejection_sample_method` `config/speculative.py:536`	`RejectionSampleMethod`	`"standard"`	The rejection sampling method to use. 'standard' uses probabilistic rejection sampling (with or without cached draft logits, controlled by draft_sample_method). 'synthetic' accepts draft tokens with a decaying probability calibrated to synthetic_acceptance_rate. 'block' uses block verification (Sun et al.), which jointly verifies the draft tokens as a block instead of one at a time. 'fly' applies …
`synthetic_acceptance_rates` `config/speculative.py:545`	`list[float] \| None`	`None`	Per-position unconditional acceptance rates for synthetic rejection sampling. Position i's entry is the marginal probability that the first i+1 draft tokens are all accepted; the list must have length num_speculative_tokens, each entry in [0, 1], and be monotonically non-increasing. Only valid when rejection_sample_method is 'synthetic'. Mutually exclusive with synthetic_acceptance_length.
`synthetic_acceptance_length` `config/speculative.py:553`	`float \| None`	`None`	Target mean acceptance length for synthetic rejection sampling, in [1, num_speculative_tokens + 1]. Resolved internally to synthetic_acceptance_rates. Only valid when rejection_sample_method is 'synthetic'. Mutually exclusive with synthetic_acceptance_rates.
`enable_adaptive_verification` `config/speculative.py:559`	`bool`	`False`	Whether to adaptively size the draft-verification budget from per-request confidence. Currently only supported for method="dspark".
`draft_sample_method` `config/speculative.py:608`	`DraftSampleMethod`	`"greedy"`	How the draft model samples tokens. 'greedy' always picks the argmax token, and the draft probabilities are treated as one-hot during rejection sampling. 'probabilistic' samples stochastically from the draft distribution and uses the full draft logits for the probability ratio test during rejection sampling. This comes at the cost of additional GPU memory usage.
`dspark_draft_topk` `config/speculative.py:616`	`int \| None`	`Field(default=None, ge=1)`	For Qwen3 DSpark drafting, evaluate the Markov projection only for the top-k base-logit candidates. Requires draft tensor parallel size 1.
`fly_window_size` `config/speculative.py:620`	`int \| None`	`Field(default=None, ge=1)`	Number of subsequent native acceptance decisions checked by FLy. If unset, defaults to ``min(6, num_speculative_tokens - 1)``.
`fly_entropy_threshold` `config/speculative.py:624`	`float`	`Field(default=0.3, ge=0)`	Target top-k entropy lower bound for FLy loose acceptance.
`fly_entropy_top_k` `config/speculative.py:627`	`int`	`Field(default=3, ge=1)`	Number of largest target probabilities used by FLy's entropy gate.

SpeechToTextConfig#

config/speech_to_text.py:55, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`sample_rate` `config/speech_to_text.py:58`	`float`	`16_000`	Sample rate (Hz) to resample input audio to. Most speech models expect 16kHz audio input. The input audio will be automatically resampled to this rate before processing.
`max_audio_clip_s` `config/speech_to_text.py:63`	`int \| None`	`30`	Maximum duration in seconds for a single audio clip without chunking. Audio longer than this will be split into smaller chunks if `allow_audio_chunking` evaluates to True, otherwise it will be rejected. `None` means audio duration can be unlimited and won't be chunked.
`overlap_chunk_second` `config/speech_to_text.py:69`	`int`	`1`	Overlap duration in seconds between consecutive audio chunks when splitting long audio. This helps maintain context across chunk boundaries and improves transcription quality at split points.
`min_energy_split_window_size` `config/speech_to_text.py:74`	`int \| None`	`1600`	Window size in samples for finding low-energy (quiet) regions to split audio chunks. The algorithm looks for the quietest moment within this window to minimize cutting through speech. Default 1600 samples ≈ 100ms at 16kHz. If None, no chunking will be done.

StructuredOutputsConfig#

config/structured_outputs.py:17, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`backend` `config/structured_outputs.py:20`	`StructuredOutputsBackend`	`"auto"`	Which engine will be used for structured outputs (e.g. JSON schema, regex, etc) by default. With "auto", we will make opinionated choices based on request contents and what the backend libraries currently support, so the behavior is subject to change in each release.
`disable_any_whitespace` `config/structured_outputs.py:25`	`bool`	`False`	If `True`, json output will always be compact without any whitespace. If `False`, the model may generate whitespace between JSON fields, which is still valid JSON. This is only supported for xgrammar and guidance backends.
`disable_additional_properties` `config/structured_outputs.py:30`	`bool`	`False`	If `True`, the `guidance` backend will not use `additionalProperties` in the JSON schema. This is only supported for the `guidance` backend and is used to better align its behaviour with `outlines` and `xgrammar`.
`reasoning_parser` `config/structured_outputs.py:34`	`str`	`""`	Select the reasoning parser depending on the model that you're using. This is used to parse the reasoning content into OpenAI API format.
`reasoning_parser_plugin` `config/structured_outputs.py:37`	`str`	`""`	Path to a dynamically reasoning parser plugin that can be dynamically loaded and registered.
`enable_in_reasoning` `config/structured_outputs.py:40`	`bool`	`False`	Whether to use structured input for reasoning.

VllmConfig#

config/vllm.py:348, decorator @config(config=ConfigDict(arbitrary_types_allowed=True)).

Field	Type	Default	Docstring (first paragraph)
`model_config` `config/vllm.py:355`	`ModelConfig`	`None`	Model configuration.
`cache_config` `config/vllm.py:357`	`CacheConfig`	`Field(default_factory=CacheConfig)`	Cache configuration.
`parallel_config` `config/vllm.py:359`	`ParallelConfig`	`Field(default_factory=ParallelConfig)`	Parallel configuration.
`scheduler_config` `config/vllm.py:361`	`SchedulerConfig`	`Field(default_factory=SchedulerConfig.default_factory)`	Scheduler configuration.
`device_config` `config/vllm.py:365`	`DeviceConfig`	`Field(default_factory=DeviceConfig)`	Device configuration.
`load_config` `config/vllm.py:367`	`LoadConfig`	`Field(default_factory=LoadConfig)`	Load configuration.
`offload_config` `config/vllm.py:369`	`OffloadConfig`	`Field(default_factory=OffloadConfig)`	Model weight offloading configuration.
`attention_config` `config/vllm.py:371`	`AttentionConfig`	`Field(default_factory=AttentionConfig)`	Attention configuration.
`aux_output_config` `config/vllm.py:373`	`AuxOutputConfig`	`Field(default_factory=AuxOutputConfig)`	Execution auxiliary output configuration.
`engram_config` `config/vllm.py:375`	`EngramConfig \| None`	`None`	N-gram embedding storage and sharding settings.
`mamba_config` `config/vllm.py:377`	`MambaConfig`	`Field(default_factory=MambaConfig)`	Mamba configuration.
`kernel_config` `config/vllm.py:379`	`KernelConfig`	`Field(default_factory=KernelConfig)`	Kernel configuration.
`lora_config` `config/vllm.py:381`	`LoRAConfig \| None`	`None`	LoRA configuration.
`speculative_config` `config/vllm.py:383`	`SpeculativeConfig \| None`	`None`	Speculative decoding configuration.
`watermark_config` `config/vllm.py:385`	`WatermarkConfig \| None`	`None`	Text watermarking configuration.
`diffusion_config` `config/vllm.py:387`	`DiffusionConfig \| None`	`None`	Diffusion LLM (dLLM) configuration.
`structured_outputs_config` `config/vllm.py:390`	`StructuredOutputsConfig`	`Field(default_factory=StructuredOutputsConfig)`	Structured outputs configuration.
`observability_config` `config/vllm.py:394`	`ObservabilityConfig`	`Field(default_factory=ObservabilityConfig)`	Observability configuration.
`logging_config` `config/vllm.py:398`	`LoggingConfig`	`Field(default_factory=LoggingConfig)`	Logging configuration.
`quant_config` `config/vllm.py:400`	`QuantizationConfig \| None`	`None`	Quantization configuration.
`compilation_config` `config/vllm.py:402`	`CompilationConfig`	`Field(default_factory=CompilationConfig)`	`torch.compile` and cudagraph capture configuration for the model.
`profiler_config` `config/vllm.py:411`	`ProfilerConfig`	`Field(default_factory=ProfilerConfig)`	Profiling configuration.
`kv_transfer_config` `config/vllm.py:413`	`KVTransferConfig \| None`	`None`	The configurations for distributed KV cache transfer.
`kv_events_config` `config/vllm.py:415`	`KVEventsConfig \| None`	`None`	The configurations for event publishing.
`ec_transfer_config` `config/vllm.py:417`	`ECTransferConfig \| None`	`None`	The configurations for distributed EC cache transfer.
`ec_manager_config` `config/vllm.py:419`	`EncoderCacheManagerConfig`	`Field(default_factory=EncoderCacheManagerConfig)`	The configurations for custom encoder cache manager.
`reasoning_config` `config/vllm.py:423`	`ReasoningConfig \| None`	`None`	The configurations for reasoning model.
`additional_config` `config/vllm.py:428`	`dict \| SupportsHash`	`Field(default_factory=dict)`	Additional config for specified platform. Different platforms may support different configs. Make sure the configs are valid for the platform you are using. Contents must be hashable.
`instance_id` `config/vllm.py:432`	`str`	`""`	The ID of the vLLM instance.
`optimization_level` `config/vllm.py:434`	`OptimizationLevel`	`OptimizationLevel.O2`	The optimization level. These levels trade startup time cost for performance, with -O0 having the best startup time and -O3 having the best performance. -O2 is used by default. See OptimizationLevel for full description.
`performance_mode` `config/vllm.py:440`	`PerformanceMode`	`"balanced"`	Performance mode for runtime behavior, 'balanced' is the default. 'interactivity' favors low end-to-end per-request latency at small batch sizes (fine-grained CUDA graphs, latency-oriented kernels). 'throughput' favors aggregate tokens/sec at high concurrency (larger CUDA graphs, more aggressive batching, throughput-oriented kernels).
`weight_transfer_config` `config/vllm.py:447`	`WeightTransferConfig \| None`	`None`	The configurations for weight transfer during RL training.
`watchdog_config` `config/vllm.py:450`	`WatchdogConfig`	`Field(default_factory=WatchdogConfig)`	Configuration for the stack-dump watchdog.
`shutdown_timeout` `config/vllm.py:453`	`int`	`Field(default=0, ge=0)`	Shutdown grace period for in-flight requests. Shutdown will be delayed for up to this amount of time to allow already-running requests to complete. Any remaining requests are aborted once the timeout is reached.

WatchdogConfig#

config/watchdog_config.py:13, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`timeout` `config/watchdog_config.py:21`	`int`	`Field(default=_DEFAULT_TIMEOUT, gt=0)`	Timeout in seconds after which the watchdog considers the process to be stuck and dumps stack traces. Must be greater than 0.
`check_interval` `config/watchdog_config.py:25`	`int`	`Field(default=_DEFAULT_CHECK_INTERVAL, gt=0)`	Interval in seconds at which the watchdog checks for feed timeouts.
`dump_dir` `config/watchdog_config.py:28`	`str`	`""`	Directory to save stack dump files. Empty string disables the watchdog feature.

WatermarkConfig#

config/watermarking.py:27, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`key` `config/watermarking.py:30`	`int`	`Field(ge=0, repr=False, exclude=True)`	Secret key used to watermark generated text.
`algorithm` `config/watermarking.py:32`	`WatermarkingAlgorithm`	`"gumbel"`	Algorithm used to watermark generated text.
`alpha` `config/watermarking.py:34`	`float`	`Field(default=0.1, ge=0, le=1)`	Probability of selecting key B for dual-key watermarking.
`context_width` `config/watermarking.py:36`	`int`	`Field(default=4, ge=1)`	Number of prior tokens used by the watermark PRF.
`deduplicate_contexts` `config/watermarking.py:38`	`WatermarkContextScope`	`"single_turn"`	Which history is searched for a repeated context before a token is sampled; a repeated context is sampled without the watermark. `none` disables the search. `single_turn` (default) searches this request's generated tokens. `all` also searches the prompt and samples the first `context_width` generated tokens without watermarking.
`deduplicate_contexts_max_history` `config/watermarking.py:44`	`int \| None`	`Field(default=8192, ge=1)`	Number of most recent history positions searched (default 8192), or `None` for the whole scope. Each position is compared over the `context_width` tokens before it. Ignored when `deduplicate_contexts` is `none`.
`prf` `config/watermarking.py:49`	`WatermarkPRFName`	`"philox"`	Pseudorandom function used by the watermarking algorithm.
`allow_target_only_watermarking` `config/watermarking.py:51`	`bool`	`False`	Allow speculative decoding without watermarking draft tokens.

WeightTransferConfig#

config/weight_transfer.py:9, decorator @config.

Field	Type	Default	Docstring (first paragraph)
`backend` `config/weight_transfer.py:12`	`Literal["nccl", "ipc", "sparse_nccl", "sharded_rdt", "nccl_m2n"] \| str`	`"nccl"`	The backend to use for weight transfer. Validated against the `WeightTransferEngineFactory` registry at engine creation time.

Check yourself#

1
You start vllm serve with --compilation-config '{"cudagraph_mod": "NONE"}' (one letter is missing). What happens?
easy
Show answer
Start-up stops with an argument error. CompilationConfig is a @config class with extra="forbid" (vllm/config/utils.py:71), so pydantic rejects the unknown key, and parse_dataclass turns the ValidationError into an argparse.ArgumentTypeError (vllm/engine/arg_utils.py:369-374).
2
Why does SchedulerConfig say max_num_seqs = 128, while vllm serve on an 80 GB H100 uses 1024?
medium
Show answer
The dataclass default is "mainly for convenience when testing" (vllm/config/scheduler.py:62-66). The flag passes None (vllm/engine/arg_utils.py:1743-1749), and EngineArgs sets the value by GPU memory before it builds SchedulerConfig (vllm/engine/arg_utils.py:2970-2983, vllm/engine/arg_utils.py:2631-2635). This assumes that the driver reports at least 70 GiB for the card (vllm/engine/arg_utils.py:2946).
3
You restart a server with --max-num-batched-tokens 16384 instead of 8192, and nothing else changes. Can it reuse the torch.compile cache of the first run?
hard
Show answer
No. SchedulerConfig.compute_hash includes max_num_batched_tokens (vllm/config/scheduler.py:280). VllmConfig.compute_hash includes the scheduler hash (vllm/config/vllm.py:500-501). The compile backend puts the config hash into the cache folder name (vllm/compilation/backends.py:1090-1097), so the new run uses a new folder and compiles again.

Config classes

The @config decorator#

Field docstrings are the help text#

VllmConfig: the container#

What __post_init__ does#

compute_hash: one key for compiled code#

How to read the tables#

All config classes#

All config classes#

What each VllmConfig attribute holds#

HiSparseConfig#

AttentionConfig#

AuxOutputConfig#

CacheConfig#

PassConfig#

DynamicShapesConfig#

CompilationConfig#

DeviceConfig#

DiffusionConfig#

EncoderCacheManagerConfig#

ECTransferConfig#

EngramConfig#

FaultToleranceConfig#

IrOpPriorityConfig#

KernelConfig#

KVEventsConfig#

KVTransferConfig#

LoadConfig#

LoggingConfig#

LoRAConfig#

MambaConfig#

ModelConfig#

MultiModalConfig#

ObservabilityConfig#

UVAOffloadConfig#

PrefetchOffloadConfig#

OffloadConfig#

EPLBConfig#

ParallelConfig#

PoolerConfig#

ProfilerConfig#

QuantSpec#

QuantizationConfigArgs#

ReasoningConfig#

SchedulerConfig#

SpeculativeConfig#

SpeechToTextConfig#

StructuredOutputsConfig#

VllmConfig#

WatchdogConfig#

WatermarkConfig#

WeightTransferConfig#

Check yourself#

Related pages

What each `VllmConfig` attribute holds#