How configuration works

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

vllm serve has about 300 flags (vllm serve flags) and more than 300 VLLM_* environment variables (Environment variables). The engine does not read the flags directly. It reads one object, the VllmConfig. The config system is the code that makes this object. It has three layers:

1. Config classes. Each part of the engine has one class in vllm/config/, for example CacheConfig or SchedulerConfig. The @config decorator makes each class a checked dataclass (Config classes).
2. EngineArgs. One flat dataclass has a field for each flag. The parser fills it (EngineArgs).
3. VllmConfig. create_engine_config() builds the sub-configs from EngineArgs and puts them into one VllmConfig. Its __post_init__ then checks them together and sets the late defaults.

Many flags have the default None. None means "vLLM decides later", from the model, the GPU, the platform or other settings. So the real default of a setting is often not the dataclass default. This page ends with a recipe to find it.

An everyday comparison. Think of an order form in a restaurant. The guest fills some boxes and leaves others empty: an empty box means "the chef decides". The head chef fills the empty boxes from the menu and the kitchen stock, and writes one final ticket. Each station gets a copy of the ticket. The comparison stops in two places. A station can write on its own copy. The copies do not go back to the head chef, except for a few values in the ready message.

The picture#

Read the panels from top to bottom. Panels 1 and 2 run only in the API server process. Panel 3 shows that the other processes do not parse flags. They get the finished object.

In the code#

Step 1: the @config decorator#

Every config class uses the decorator config (vllm/config/utils.py:51-61). It wraps the class in a pydantic dataclass with extra="forbid":

    # Extra fields are forbidden by default
    merged_config = ConfigDict(extra="forbid")
    if config is not None:
        merged_config.update(config)

    def decorator(cls: type[ConfigT]) -> type[ConfigT]:
        return dataclass(cls, config=merged_config, **kwargs)  # type: ignore[return-value]

    # Called with arguments: @config(config=...)
    if cls is None:
        return decorator
    # Called without arguments: @config
    return decorator(cls)

1. Line 71. An unknown key is an error. So -cc '{"mod": 3}' stops start-up, and a typo does not pass in silence.
2. Lines 75-76. dataclass is pydantic.dataclasses.dataclass (vllm/config/utils.py:20). Pydantic checks the types and limits such as Field(ge=1) when the object is made.

Each field has a docstring, a string literal on the line after the field. The parser uses these strings as the --help text. The page Config classes lists every class and field; this page does not repeat the tables.

Step 2: VllmConfig holds the top-level configs#

@config(config=ConfigDict(arbitrary_types_allowed=True))
class VllmConfig:
    """Dataclass which contains all vllm-related configuration. This
    simplifies passing around the distinct configurations in the codebase.
    """

    # TODO: use default_factory once default constructing ModelConfig doesn't
    # try to download a model
    model_config: ModelConfig = None  # type: ignore[assignment]
    """Model configuration."""
    cache_config: CacheConfig = Field(default_factory=CacheConfig)
    """Cache configuration."""
    parallel_config: ParallelConfig = Field(default_factory=ParallelConfig)
    """Parallel configuration."""
    scheduler_config: SchedulerConfig = Field(
        default_factory=SchedulerConfig.default_factory,
    )
    """Scheduler configuration."""
    device_config: DeviceConfig = Field(default_factory=DeviceConfig)
    """Device configuration."""

VllmConfig has 34 attributes (vllm/config/vllm.py:355-453). Most use a default factory, so a plain VllmConfig() makes default sub-configs. model_config is None by default, because a default ModelConfig would download a model (comment at lines 353-354). Optional features, for example lora_config and speculative_config, are None when they are off (vllm/config/vllm.py:381-384). Model code reads the config through get_current_vllm_config(), inside a set_current_vllm_config() block (vllm/config/vllm.py:3516-3540).

Step 3: EngineArgs, one flat dataclass#

EngineArgs has one field for each flag. Most fields copy their default from a config class:

@dataclass
class EngineArgs:
    """Arguments for vLLM engine."""

    model: str = ModelConfig.model
    # Public compatibility argument. Canonical runtime state lives in
    # AuxOutputConfig.
    enable_return_routed_experts: bool = False
    return_sampling_mask: bool = ModelConfig.return_sampling_mask
    model_weights: str = ModelConfig.model_weights
    served_model_name: str | list[str] | None = ModelConfig.served_model_name
    tokenizer: str | None = ModelConfig.tokenizer
    hf_config_path: str | None = ModelConfig.hf_config_path
    runner: RunnerOption = ModelConfig.runner
    convert: ConvertOption = ModelConfig.convert
    skip_tokenizer_init: bool = ModelConfig.skip_tokenizer_init
    enable_prompt_embeds: bool = ModelConfig.enable_prompt_embeds
    tokenizer_mode: TokenizerMode | str = ModelConfig.tokenizer_mode

A field with a default factory uses get_field, which copies the factory of the config field (vllm/config/utils.py:85-114). Some fields do not copy a default. They start as None, so that create_engine_config can decide:

    block_size: int | None = None
    enable_prefix_caching: bool | None = None

    max_num_batched_tokens: int | None = None
    max_num_scheduled_tokens: int | None = None
    long_prefill_token_threshold: int = SchedulerConfig.long_prefill_token_threshold
    long_prefill_token_threshold_adaptive: bool = (
        SchedulerConfig.long_prefill_token_threshold_adaptive
    )
    max_num_seqs: int | None = None

enable_chunked_prefill is also None (vllm/engine/arg_utils.py:707). EngineArgs.__post_init__ turns dict values into config objects, for example a compilation_config dict, and loads the general plugins (vllm/engine/arg_utils.py:873-924).

Step 4: add_cli_args makes the flags from the config classes#

EngineArgs.add_cli_args makes one argument group for most config classes. Each group has the class name as its title and the class docstring as its description:

        # Model arguments
        model_kwargs = get_kwargs(ModelConfig)
        aux_output_kwargs = get_kwargs(AuxOutputConfig)
        model_group = parser.add_argument_group(
            title="ModelConfig",
            description=ModelConfig.__doc__,
        )
        if not ("serve" in sys.argv[1:] and "--help" in sys.argv[1:]):
            model_group.add_argument("--model", **model_kwargs["model"])

The flag name is written by hand. All other argparse settings come from get_kwargs(ModelConfig)["model"]. get_kwargs returns a deep copy of a cached result (vllm/engine/arg_utils.py:443-453). The cached function reads each dataclass field (vllm/engine/arg_utils.py:324-440):

• Default: the field default, or the result of its default factory (lines 339-352).
• Help: the field docstring. The parser reads the docstrings only when the command line contains --help, because that is slow (vllm/engine/arg_utils.py:285-288, vllm/engine/arg_utils.py:327).
• JSON: a field whose type is a dataclass takes a JSON string, which pydantic checks (lines 367-378).
• Booleans: a bool field gets BooleanOptionalAction (lines 384-386). FlexibleArgumentParser replaces it with EnableDisableAction, which pairs --enable-x with --disable-x (vllm/utils/argparse_utils.py:120-145, vllm/utils/argparse_utils.py:179).

Some flags replace the field default with None, for example --max-num-seqs (vllm/engine/arg_utils.py:1743-1749). The VllmConfig group holds the JSON flags of whole sub-configs, for example --compilation-config and its short form -cc (vllm/engine/arg_utils.py:1880-1882).

Step 5: the parser rewrites the command line#

FlexibleArgumentParser.parse_args changes the list of arguments before argparse sees it:

        if "--config" in args:
            args = self._pull_args_from_config(args)

        def repl(match: re.Match) -> str:
            """Replaces underscores with dashes in the matched string."""
            return match.group(0).replace("_", "-")

        # Everything between the first -- and the first .
        pattern = re.compile(r"(?<=--)[^\.]*")

        # Convert underscores to dashes and vice versa in argument names
        processed_args = list[str]()
        for i, arg in enumerate(args):
            if arg.startswith("--help="):
                FlexibleArgumentParser._search_keyword = arg.split("=", 1)[-1].lower()
                processed_args.append("--help")
            elif arg.startswith("--"):
                if "=" in arg:
                    key, value = arg.split("=", 1)
                    key = pattern.sub(repl, key, count=1)
                    processed_args.append(f"{key}={value}")
                else:
                    key = pattern.sub(repl, arg, count=1)
                    processed_args.append(key)
            elif arg.startswith("-O") and arg != "-O":
                # allow -O flag to be used without space, e.g. -O3 or -Odecode
                # also handle -O=<optimization_level> here
                optimization_level = arg[3:] if arg[2] == "=" else arg[2:]
                processed_args += ["--optimization-level", optimization_level]
            elif (
                arg == "-O"
                and i + 1 < len(args)
                and args[i + 1] in {"0", "1", "2", "3"}
            ):
                # Convert -O <n> to --optimization-level <n>
                processed_args.append("--optimization-level")
            else:

1. Lines 371-372. --config file.yaml inserts the keys of a YAML file as flags (next step).
2. Lines 379-394. In a flag name, _ becomes -, up to the first dot. So --max_num_seqs works. Keys after a dot keep their underscores.
3. Lines 395-406. -O3, -O=3 and -O 3 become --optimization-level 3.

Then a flag with a dot becomes a JSON value: -cc.mode=3 becomes -cc '{"mode": 3}'. Several dotted flags with the same prefix merge into one dict (vllm/utils/argparse_utils.py:462-500). A key that ends with + appends list items (lines 472-474). The help text of the JSON flags shows these forms (vllm/utils/argparse_utils.py:159-167).

Config files. The parser supports YAML only: another file extension raises ValueError (vllm/utils/argparse_utils.py:633-637). A true value becomes --key, a list becomes --key v1 v2, and a nested dict becomes a JSON string (vllm/utils/argparse_utils.py:653-671). The file arguments go right after serve and the model. So a flag on the command line wins: "cli > config > defaults" (vllm/utils/argparse_utils.py:563-600).

Step 6: create_engine_config, step by step#

The API server calls AsyncEngineArgs.from_cli_args(args). It copies each parsed value into the field of the same name (vllm/engine/arg_utils.py:1961-1970). Then it calls create_engine_config() with the usage context OPENAI_API_SERVER (vllm/entrypoints/launchers/api_server/entry.py:52, vllm/entrypoints/launchers/api_server/entry.py:82). The function starts like this:

        logging_config = self.create_logging_config()
        configure_logging_if_needed(logging_config)

        current_platform.pre_register_and_update()

        device_config = DeviceConfig(device=cast(Device, current_platform.device_type))

        current_platform.validate_environ(self.fail_on_environ_validation)

The order of the work matters, because later steps read the results of earlier ones:

Step 2 can take time on a cold start. ModelConfig loads the Hugging Face config and asks the model registry, which can start a subprocess on a cache miss (Out-of-tree models and the model registry). It logs "Resolved architecture: ..." (vllm/config/model.py:741-744).

Step 7: defaults from the usage context and the GPU#

_set_default_max_num_seqs_and_batched_tokens_args fills the two batch limits when they are None:

        if self.max_num_batched_tokens is None:
            if parallel_config.use_batched_dp_moe:
                self.max_num_batched_tokens = (
                    SchedulerConfig.DEFAULT_MAX_NUM_BATCHED_TOKENS_FOR_BATCHED_DP
                )
            else:
                self.max_num_batched_tokens = default_max_num_batched_tokens.get(
                    usage_context,
                    SchedulerConfig.DEFAULT_MAX_NUM_BATCHED_TOKENS,
                )

        if self.max_num_seqs is None:
            self.max_num_seqs = default_max_num_seqs.get(
                usage_context,
                SchedulerConfig.DEFAULT_MAX_NUM_SEQS,
            )

        # If throughput mode is set, double max_num_batched_tokens and max_num_seqs.
        if self.performance_mode == "throughput":
            if orig_max_num_batched_tokens is None:
                self.max_num_batched_tokens *= 2
            if orig_max_num_seqs is None:
                self.max_num_seqs *= 2

get_batch_defaults gives one dict per limit, keyed by the usage context (vllm/engine/arg_utils.py:2926-3026). Take OPENAI_API_SERVER on a GPU with 70 GiB or more and less than 160 GiB. If it is not an A100 or GB10, the values are 8192 tokens and 1024 requests (vllm/engine/arg_utils.py:2970-2983). A usage context that is not in the dict falls back to the SchedulerConfig constants 2048 and 128 (vllm/config/scheduler.py:41-43). The table vllm serve flags lists all tiers.

Then caps apply, only to values that you did not set. Without chunked prefill, the budget is at least max_model_len tokens. The budget is at most max_num_seqs × max_model_len tokens, and the request limit is at most the budget (vllm/engine/arg_utils.py:3177-3221). A multimodal prefix-LM model can also raise the budget, so that one multimodal item fits in one step (lines 3191-3202).

Step 8: VllmConfig.__post_init__, the late defaults#

The __post_init__ of VllmConfig is about 830 lines long (vllm/config/vllm.py:1492-2318). Its phases, in code order:

1. Model hooks. try_verify_and_update_config runs the per-architecture config class of the model, once (vllm/config/vllm.py:1521, vllm/config/vllm.py:2714-2743).
2. Quantization. It builds quant_config from the model (vllm/config/vllm.py:1586-1589).
3. Async scheduling. When async_scheduling is None, it turns it on unless a pooling model, the executor or another option conflicts (vllm/config/vllm.py:1664-1698). See Async scheduling: overlap the CPU and the GPU.
4. Eager mode. --enforce-eager sets the compilation mode and the CUDA graph mode to NONE (vllm/config/vllm.py:1745-1747).
5. Platform defaults and compilation. apply_config_platform_defaults, then the compilation mode from the optimization level, then the defaults table of that level (vllm/config/vllm.py:1804-1810, vllm/config/vllm.py:1833-1834).
6. CUDA graph sizes on platforms with static graphs (vllm/config/vllm.py:1929-1995).
7. The platform's last word. current_platform.check_and_update_config(self) (vllm/config/vllm.py:2046). CUDA sets the worker class here (vllm/platforms/cuda.py:330-331).
8. Model runner checks and compile ranges (vllm/config/vllm.py:2071-2083).
9. KV transfer and the hybrid KV cache manager (vllm/config/vllm.py:2199, vllm/config/vllm.py:2236-2273).

The defaults of an optimization level fill only fields that are still None. A value that you set is never overwritten:

        if getattr(config_obj, key) is None:
            # Some config values are known before initialization and are
            # hard coded.
            # Other values depend on the user given configuration, so they are
            # implemented with lambda functions and decided at run time.
            setattr(config_obj, key, value(self) if callable(value) else value)

So -O1 -cc.cudagraph_mode=FULL_AND_PIECEWISE keeps your graph mode, and takes the other O1 defaults (vllm/config/vllm.py:275-295). The page config/vllm.py: VllmConfig, __post_init__ and compute_hash walks through every phase.

Step 9: how the config reaches the other processes#

The front end does not send flags to the EngineCore. It sends the finished VllmConfig object as an argument of the new process:

        context = get_mp_context()
        common_kwargs = {
            "vllm_config": vllm_config,
            "local_client": local_client,
            "handshake_address": handshake_address,
            "executor_class": executor_class,
            "log_stats": log_stats,
            "tensor_queue": tensor_queue,
        }

The CLI selects the spawn start method (vllm/entrypoints/common/startup.py:147-149). With spawn, Python pickles the process arguments at start(); a comment in the same class relies on this (vllm/v1/engine/utils.py:295-298). The executor does the same for each worker process (vllm/v1/executor/multiproc_executor.py:731-750). With TP = 1 there is no worker process: the worker object lives in the engine core and uses its copy (vllm/v1/executor/uniproc_executor.py:54-68).

Values do not flow back, with one exception. The ready response of each engine core carries values that it found at start-up. The front end copies max_model_len (the smaller one), num_gpu_blocks (the sum over engines) and block_size into its own config (vllm/v1/engine/core_client.py:847-879).

Step 10: compute_hash, one key for compiled code#

VllmConfig.compute_hash() collects the hash of each sub-config and adds the vLLM version. It keeps the first 10 hex digits of an MD5 hash (SHA-256 in FIPS mode):

        factors.append(vllm_factors)

        hash_str = safe_hash(str(factors).encode(), usedforsecurity=False).hexdigest()[
            :10
        ]
        return hash_str

Each sub-config decides which fields count. The docstring says that the hash covers the configs "that affect the structure of the computation graph" (vllm/config/vllm.py:464-468). SchedulerConfig counts max_num_batched_tokens and max_num_seqs, because compiled graphs capture buffer sizes that depend on them (vllm/config/scheduler.py:271-284). Three users of the hash:

• The compile cache folder name, when compilation_config.cache_dir is empty: a SHA-256 of the env hash, the config hash, the code hash and the compiler hash (vllm/compilation/backends.py:1059-1063, vllm/compilation/backends.py:1086-1098).
• The AOT compile cache, keyed by the env hash and the config hash (vllm/compilation/caching.py:572-588).
• The optional start-up memory plan of VLLM_ENABLE_STARTUP_PLAN (vllm/v1/worker/startup_plan.py:61-74).

So a change of --max-num-seqs gives a new hash, and the next start compiles again. The page torch.compile and CUDA graphs in vLLM explains the cache.

Step 11: environment variables and flags#

Environment variables are a separate channel. They are not fields of VllmConfig. vllm/envs.py maps each name to a function, mostly a lambda (vllm/envs.py:618). A read of envs.X calls the module __getattr__, which runs the function again:

def __getattr__(name: str):
    """Gets environment variables lazily.

    NOTE: After enable_envs_cache() invocation (which triggered after service
    initialization), all environment variables will be cached.
    """
    if name in environment_variables:
        return environment_variables[name]()
    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

The environment variable cache stops this. enable_envs_cache() runs at the end of EngineCore.__init__ and at the end of WorkerProc.__init__ (vllm/v1/engine/core.py:241-243, vllm/v1/executor/multiproc_executor.py:708-710). No front-end code calls it, so the API server reads the environment live. The page envs.py: environment variables as a dict of lambdas reads the module, and Environment variables lists every variable.

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

vllm serve Qwen/Qwen3-0.6B --config serve.yaml --max_model_len 4096 -O1 -cc.cudagraph_mode=PIECEWISE

What can go wrong#

• A typo in a JSON key. extra="forbid" rejects it, and argparse stops with the pydantic error (vllm/engine/arg_utils.py:369-374). This is a feature: fix the key.
• A typo in an environment variable. It is only a warning, unless you add --fail-on-environ-validation (vllm/engine/arg_utils.py:1922-1928).
• --config=serve.yaml. The parser expands the file only when --config is a separate word (vllm/utils/argparse_utils.py:371-372). No code in vllm/ reads the value of args.config, so the file is ignored with no error. Write --config serve.yaml. (Derived from a search of the repository.)
• A file with another extension. serve.json raises "Config file must be of a yaml/yml type" (vllm/utils/argparse_utils.py:633-637).
• A change of the environment after start-up. It does not reach the engine core or the workers: they inherited the environment at spawn and cache it after start-up.
• An unexpected recompile. A change of --max-num-seqs, --max-num-batched-tokens or most VLLM_* variables gives a new cache key (Step 10).

Why it matters. To tune a server, you must know the value that the engine really uses. The value can come from a flag, the model, the GPU, the platform or the engine core. The config system decides it in a fixed order, and only in the front end, except for the steps in the engine core.