Tuning vLLM: throughput and latency knobs, their real defaults and what each one does

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

A vLLM server has a few dozen flags that change speed. They act in three places, at three times. This page sorts them by place, gives each real default with the line that sets it, and says what the knob trades. It does not give performance numbers: measure your own workload with Benchmarking vLLM: vllm bench serve, throughput, latency and startup.

1. Memory, once at start-up. --gpu-memory-utilization, --kv-cache-memory-bytes, --kv-cache-dtype, --max-model-len and the size of the weights decide how many KV blocks exist. The block count does not change while the server runs. The engine core logs it as the maximum concurrency.
2. The scheduler, every step. --max-num-seqs, --max-num-batched-tokens (the token budget), chunked prefill and the scheduling policy decide which requests share a step. They trade time to first token (TTFT) against inter-token latency (ITL) and throughput (Serving metrics: TTFT, TPOT, ITL, throughput, goodput).
3. Execution and the front end. CUDA graphs, torch.compile, speculative decoding and async scheduling change how long a step takes. --api-server-count, the renderer threads and --stream-interval change the CPU work in the API server.

Most knobs have a default that the code computes. The dataclass default is often not the real default: for example, SchedulerConfig says 128 sequences, but vllm serve on an 80 GiB H100 uses 1024. The full flag list with real defaults is on the serve-flags reference; this page explains the mechanisms behind the most important ones.

An everyday comparison. Think of a restaurant. The size of the dining room is fixed when it opens (memory knobs). The head waiter decides each minute which guests to seat and how many dishes the kitchen starts (scheduler knobs). The kitchen equipment and the number of waiters at the door decide how fast one round goes (execution and front-end knobs). The comparison stops at the tables. In vLLM, one long conversation can take the space of many short ones. A guest can also lose the table (preemption) when the room is full.

The picture#

Read the panels from top to bottom. Panel 1 runs once, so its knobs need a restart. Panel 2 runs in the EngineCore process at every step. Panel 3 mixes the GPU workers (compile, graphs, drafts) and the API server process (amber).

In the code#

Step 1: the memory budget and the KV cache size#

At start-up, each worker measures how much memory is left for the KV cache. The page The KV cache manager: blocks, the block pool and memory sizing explains the block pool; Executors and workers: from the engine core to the GPUs explains the profile run. The worker first computes the requested memory, a share of the whole GPU, and refuses to start when less is free (vllm/v1/worker/utils.py:724-726, vllm/v1/worker/utils.py:761-770). After a dummy forward pass with max_num_batched_tokens tokens, it computes the rest:

        self.total_consumed = profile_result.total_consumed
        self.peak_activation_memory = (
            profile_result.transient_peak_headroom + cudagraph_memory_estimate_applied
        )
        self.cudagraph_memory_estimate = cudagraph_memory_estimate

        self.available_kv_cache_memory_bytes = (
            self.requested_memory
            - profile_result.non_kv_cache_memory
            - cudagraph_memory_estimate_applied
        )

non_kv_cache_memory is the memory consumed since the start snapshot (weights, buffers, non-torch memory) plus the transient peak of the activations (vllm/utils/mem_utils.py:312-321). The CUDA graph estimate is on by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1, vllm/envs.py:2217-2219). So the KV cache gets:

available_kv = ceil(total × gpu_memory_utilization) − (consumed + peak_activation_headroom) − cudagraph_estimate

Four knobs change this result:

• --gpu-memory-utilization, default 0.92, is the share of the whole GPU for this instance (vllm/config/cache.py:212-221). Each 0.01 adds 1 % of the GPU to the KV cache.
• --max-num-batched-tokens sets the size of the profile run, so a larger budget measures a larger activation peak and leaves less KV memory (vllm/v1/worker/gpu/model_runner.py:231, vllm/v1/worker/gpu/model_runner.py:1022-1027).
• --enforce-eager turns off CUDA graphs, so the graph estimate is 0 and no graph is captured (vllm/v1/worker/gpu_worker.py:704-708, vllm/v1/worker/gpu_worker.py:983-985).
• --kv-cache-memory-bytes replaces the whole formula. The worker still runs the profile run (it compiles the model), but then returns the given size and logs that it does not respect gpu_memory_utilization (vllm/v1/worker/gpu_worker.py:653-679).

Step 2: max_model_len and the maximum-concurrency line#

After the KV cache is sized, the engine core stores the capacity and logs one line (vllm/v1/engine/core.py:342):

def update_kv_cache_capacity(
    vllm_config: VllmConfig, kv_cache_config: KVCacheConfig
) -> None:
    """Store and log the resolved KV cache capacity."""
    num_tokens, max_concurrency = get_kv_cache_capacity(vllm_config, kv_cache_config)
    vllm_config.cache_config.kv_cache_size_tokens = num_tokens
    vllm_config.cache_config.kv_cache_max_concurrency = max_concurrency
    max_model_len = vllm_config.model_config.max_model_len
    pools = _kv_cache_device_name(vllm_config)
    if vllm_config.attention_config.hisparse_config is not None:
        pools += " + host"
    logger.info_once(
        "%s KV cache size: %s tokens, "
        "Maximum concurrency for %s tokens per request: %.2fx",
        pools,
        f"{num_tokens:,}",
        f"{max_model_len:,}",
        max_concurrency,
    )

The concurrency is the number of blocks divided by the blocks that one request of max_model_len needs. For a model with one KV cache group, that is num_blocks / cdiv(max_model_len, block_size). With several groups, vLLM adds the blocks per request of all groups (vllm/v1/core/kv_cache_utils.py:1031-1041). The token count in the log is int(concurrency × max_model_len) (vllm/v1/core/kv_cache_utils.py:1074-1082). So "Maximum concurrency 3.41x" means: 3.41 requests of the full length fit at once. Shorter requests fit many more times.

--max-model-len does not change the block count. It changes the check at start-up: one request of that length must fit in the KV cache, minus one block for the null block (vllm/v1/core/kv_cache_planning.py:1572-1582, vllm/v1/core/kv_cache_planning.py:1590-1598). If it does not fit, start-up stops with an estimate of the largest length that fits (vllm/v1/core/kv_cache_planning.py:68-87). With --max-model-len auto (or -1), vLLM lowers the length to that estimate and logs "Auto-fit max_model_len: reduced from ..." (vllm/v1/core/kv_cache_planning.py:1584-1587, vllm/v1/core/kv_cache_planning.py:1393-1410). The default is the length from the model's config (vllm/config/model.py:2639-2653).

The playground uses only these formulas from the code:

1. requested = ceil(total × gpu_memory_utilization) (vllm/v1/worker/utils.py:724-726).
2. available = requested − (consumed + peak_headroom) − cudagraph_estimate (vllm/v1/worker/gpu_worker.py:744-748, vllm/utils/mem_utils.py:312-321). With --kv-cache-memory-bytes: available = that value (vllm/v1/worker/gpu_worker.py:653-679).
3. bytes_per_block = bytes_per_token × block_size, where bytes per token is the sum over the layers of num_kv_heads × (head_size + head_size_v) × dtype_bytes (vllm/v1/kv_cache_interface.py:571-579, vllm/v1/core/kv_cache_planning.py:587-593).
4. num_blocks = available // bytes_per_block (vllm/v1/core/kv_cache_planning.py:1610-1611).
5. Fit check: cdiv(max_model_len, block_size) × bytes_per_block ≤ available − bytes_per_block (vllm/v1/core/kv_cache_planning.py:1572-1582, vllm/v1/core/kv_cache_planning.py:68-87). When available − bytes_per_block ≤ 0, start-up stops earlier with "No available memory for the cache blocks." (vllm/v1/core/kv_cache_planning.py:56-64).
6. max_concurrency = num_blocks / cdiv(max_model_len, block_size); logged tokens = int(max_concurrency × max_model_len) (vllm/v1/core/kv_cache_utils.py:1031-1047, vllm/v1/core/kv_cache_utils.py:1074-1082).
7. "Use vllm serve defaults" applies the memory tiers, the throughput doubling and the two caps of Step 3 (vllm/engine/arg_utils.py:2958-2993, vllm/engine/arg_utils.py:3171-3175, vllm/engine/arg_utils.py:3206-3221).

Step 3: max_num_seqs and max_num_batched_tokens#

Both flags are None by default. create_engine_config fills them by the usage context and the memory of one GPU (usage context). For vllm serve (OPENAI_API_SERVER):

        # throughput, see PR #17885 for more details.
        # So here we do an extra device name check to prevent such regression.
        if device_memory >= 160 * GiB_bytes:
            # for GPUs like B200/B300 with >= 160GB memory, use the largest defaults
            default_max_num_batched_tokens = {
                UsageContext.LLM_CLASS: 16384,
                UsageContext.OPENAI_API_SERVER: 16384,
            }
            default_max_num_seqs = {
                UsageContext.LLM_CLASS: 1024,
                UsageContext.OPENAI_API_SERVER: 1024,
            }
        elif (
            device_memory >= 70 * GiB_bytes
            and "a100" not in device_name
            and not is_gb10
        ):
            # For GPUs like H100 and H200, use larger offline defaults.
            default_max_num_batched_tokens = {
                UsageContext.LLM_CLASS: 16384,
                UsageContext.OPENAI_API_SERVER: 8192,
            }
            default_max_num_seqs = {
                UsageContext.LLM_CLASS: 1024,
                UsageContext.OPENAI_API_SERVER: 1024,
            }
        else:
            # TODO(woosuk): Tune the default values for other hardware.
            default_max_num_batched_tokens = {
                UsageContext.LLM_CLASS: 8192,
                UsageContext.OPENAI_API_SERVER: 2048,
            }
            default_max_num_seqs = {
                UsageContext.LLM_CLASS: 256,
                UsageContext.OPENAI_API_SERVER: 256,
            }

Then these rules apply, only to values that you did not set yourself:

        # If throughput mode is set, double max_num_batched_tokens and max_num_seqs.
        if self.performance_mode == "throughput":
            if orig_max_num_batched_tokens is None:
                self.max_num_batched_tokens *= 2
            if orig_max_num_seqs is None:
                self.max_num_seqs *= 2

• --performance-mode throughput doubles both defaults (above). The other modes are balanced (default) and interactivity (vllm/config/vllm.py:146).
• Without chunked prefill, the token budget is at least max_model_len (vllm/engine/arg_utils.py:3177-3186).
• The token budget is at most max_num_seqs × max_model_len (vllm/engine/arg_utils.py:3206-3211).
• max_num_seqs is at most the token budget (vllm/engine/arg_utils.py:3219-3221).
• For a multimodal prefix-LM model, the budget grows until the largest multimodal item fits (vllm/engine/arg_utils.py:3188-3203).

Values that you set are checked, not changed. max_num_seqs larger than the token budget raises ValueError at start-up (vllm/config/scheduler.py:331-336). When chunked prefill is on, SchedulerConfig logs the budget once: "Chunked prefill is enabled with max_num_batched_tokens=N." (vllm/config/scheduler.py:309-313).

What the two knobs change:

• max_num_seqs caps the running requests per step (vllm/v1/core/sched/scheduler.py:979-981). It also sizes the per-request buffers of the model runner and the default CUDA graph capture sizes (Step 6). A larger value gives more throughput only when the KV cache can hold that many requests. --max-num-active-seqs lowers only the admission limit and keeps the buffers (vllm/config/scheduler.py:69-76).
• max_num_batched_tokens caps the tokens per step. The scheduler gives running requests their tokens first, then admits waiting requests with the rest (vllm/v1/core/sched/scheduler.py:713-715, vllm/v1/core/sched/scheduler.py:956-957). A larger budget moves prompts through faster (lower TTFT under load) but makes the steps longer, so running requests see a higher ITL. It also sets the profile-run size (Step 1) and is part of the compile cache key, so a change compiles again (vllm/config/scheduler.py:271-284).

Step 4: chunked prefill and the long-prefill cap#

Chunked prefill is on for generative decoder models: the default comes from is_chunked_prefill_supported of the model, not from the dataclass (vllm/engine/arg_utils.py:3031-3040). A prompt that does not fit the rest of the budget gets a chunk, and the next steps continue it (Batching↗). Without chunked prefill, the budget must be at least max_model_len (vllm/config/scheduler.py:318-329).

A long prompt can still take the whole rest of a step. --long-prefill-token-threshold N caps the tokens that one prefill takes per step. The default 0 turns the cap off (vllm/config/scheduler.py:79-84):

        # `long_prefill_token_threshold` exists to stop a long prefill from
        # starving other requests of the token budget. When it is the only
        # request there is nobody to starve, so let it use the whole budget.
        num_running, num_waiting = self.get_request_counts()
        num_eligible_reqs = num_running + num_waiting
        long_prefill_token_threshold = (
            self.scheduler_config.long_prefill_token_threshold
            if num_eligible_reqs > 1
            else 0
        )
        if long_prefill_token_threshold > 0 and self.adaptive_long_prefill_threshold:
            # Floor the cap at a fair share of the input budget so it never
            # cuts a request below max_num_batched_tokens / num requests.
            long_prefill_token_threshold = max(
                long_prefill_token_threshold, input_budget // num_eligible_reqs
            )

The cap applies only when more than one request runs or waits (lines 679-683). With --long-prefill-token-threshold-adaptive, the cap is never below a fair share of the budget (lines 684-689). A smaller cap spreads a long prompt over more steps: its own TTFT grows, and the other requests get more room in each step.

Step 5: bytes per token: block size, prefix caching, KV dtype, quantization#

• --block-size: 16 tokens unless an attention backend needs another size (vllm/config/cache.py:182; Attention backends: backend, metadata builder and implementation). The block size does not change the bytes per token. It changes the rounding: a request uses cdiv(tokens, block_size) blocks, and only full blocks are cached for prefix hits.
• --enable-prefix-caching: on by default for generative decoder and hybrid models (vllm/engine/arg_utils.py:3062-3063). Requests with a shared prefix share blocks and skip that part of the prefill (Prefix caching in the engine core: hashes, lookup, caching, eviction and KV events). It changes no block count, but more requests fit when they share blocks.
• --kv-cache-dtype fp8: 1 byte per value instead of 2, so the page size and the bytes per token halve (derived from vllm/v1/kv_cache_interface.py:571-575). The default auto uses the model data type (vllm/config/cache.py:232-233), unless the checkpoint declares a KV cache scheme, for example an FP8 one (vllm/utils/torch_utils.py:503-521, vllm/model_executor/layers/attention/attention.py:287-291). Without scales in the checkpoint, the layer uses the scale 1.0 (vllm/model_executor/layers/attention/attention.py:133-154). Check the output quality of your model after the change (Number formats↗).
• --quantization: smaller weights leave more memory for the KV cache. The default is the quantization_config of the checkpoint (vllm/config/model.py:264-268); see Quantized checkpoints and Quantization methods as plugins.

Step 6: compile, CUDA graphs and -O levels: start-up time against step time#

The default optimization level is O2 (vllm/config/vllm.py:434). O2 compiles the model with torch.compile and captures FULL and PIECEWISE CUDA graphs (vllm/config/vllm.py:131-143, vllm/config/vllm.py:310). torch.compile and CUDA graphs in vLLM explains both. For tuning, the trade is simple: compile and capture cost start-up time and some memory, and save launch overhead in every step (CUDA graphs↗).

• -O0 or --enforce-eager: no compile, no graphs. --enforce-eager sets the compile mode and the graph mode to NONE (vllm/config/vllm.py:1745-1747). Start-up is fastest, and the graph memory goes to the KV cache (Step 1). Every step pays the full kernel launch cost.
• -O1: compile with PIECEWISE graphs only.
• Capture sizes. When you set no sizes, the largest captured batch is min(max_num_seqs × decode_query_len × 2, 512) (1024 on data-center Blackwell), then at most the token budget (vllm/config/vllm.py:2439-2449, vllm/config/vllm.py:2530-2531). The sizes are 1, 2, 4, then steps of 8 up to 248, then steps of 16 (vllm/config/vllm.py:2557-2570). A decode batch larger than the largest size runs without a graph.
• --performance-mode interactivity captures every size from 1 to 32 instead of 1, 2, 4, 8, 16, 24, 32 (vllm/config/vllm.py:2552-2556). Small batches then need less padding, and start-up captures more graphs.

The warm start matters as much as the level: the compile cache under VLLM_CACHE_ROOT makes the second start-up much shorter (Tour: Start-up of vllm serve, Troubleshooting vLLM: symptom, cause and fix).

Step 7: async scheduling and speculative decoding: fewer idle gaps, fewer steps#

Async scheduling is on by default for generative models when the executor supports it (vllm/config/vllm.py:1664-1698). The engine core schedules step N+1 while the GPU runs step N (Async scheduling: overlap the CPU and the GPU). Turn it off (--no-async-scheduling) only to compare.

Speculative decoding (--speculative-config, off by default, vllm/config/vllm.py:383) proposes k draft tokens and checks them in one target step (Speculative decoding↗; Speculative decoding in vLLM serving). It saves steps only when drafts are accepted. The vLLM docs say that it reduces "inter-token latency under medium-to-low QPS (queries per second), memory-bound workloads" (docs/features/speculative_decoding/README.md:3). Their table rates the gain per method for low and high QPS (docs/features/speculative_decoding/README.md:30-39). Watch the acceptance with these PromQL formulas from the code (vllm/v1/spec_decode/metrics.py:178-196):

acceptance rate      = rate(vllm:spec_decode_num_accepted_tokens_total[5m]) / rate(vllm:spec_decode_num_draft_tokens_total[5m])
mean acceptance len  = 1 + rate(vllm:spec_decode_num_accepted_tokens_total[5m]) / rate(vllm:spec_decode_num_drafts_total[5m])

The log line "SpecDecoding metrics: Mean acceptance length: ..." shows the same values every log interval (vllm/v1/spec_decode/metrics.py:107-136). A mean acceptance length near 1 means that the drafts cost work and save no step.

Step 8: parallelism, API servers, streaming and priorities#

• TP or DP. Tensor parallelism splits one model copy over several GPUs: it fits larger models and makes each step faster, with an all-reduce per layer (Tensor and pipeline parallelism in practice). Data parallelism (-dp N) runs N engine copies; each has its own scheduler and KV cache (Data parallel serving: several engine cores behind one endpoint). A practical rule, not a measured number: when one copy fits on one GPU, more copies give more throughput. TP gives a lower latency per request.
• --api-server-count runs N API server processes on one port. The default is the DP size with internal load balancing, the local DP size with hybrid load balancing, else 1 (vllm/entrypoints/cli/serve.py:113-130). Rendering, tokenization, detokenization and JSON run in the API server (The OpenAI-compatible server: launch modes, app, serving classes and streaming). When the API server process is at 100 % CPU, more API servers help; the vLLM docs show --api-server-count 4 with one engine (docs/configuration/optimization.md:316-327). With more than one API server, the periodic log line is off and Prometheus uses a multiprocess directory (vllm/v1/metrics/loggers.py:1385-1389, vllm/entrypoints/cli/serve.py:282-283).
• --renderer-num-workers, default 1, is the number of threads that tokenize and render chat templates in each API server (vllm/config/model.py:412-420, vllm/renderers/base.py:143-144).
• --stream-interval N, default 1, makes the front end send a stream chunk only every N tokens, plus the first and the last (vllm/config/scheduler.py:216-220, vllm/v1/engine/output_processor.py:379-391). Fewer chunks mean less CPU work per token. A request can raise it with its own stream_interval field, never lower it (vllm/v1/engine/output_processor.py:300-302).
• --scheduling-policy priority admits requests by the priority field (lower value first) and preempts the request with the largest priority value first (vllm/config/scheduler.py:159-165; The V1 scheduler: one token budget, no phases). The header X-Vllm-Priority overrides the body field (vllm/entrypoints/generate/base/serving.py:253-265). With the default fcfs, the field has no effect.

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

What can go wrong#

The table maps a symptom to the knob to try first. Each row is advice; the cause column cites the code.

Why it matters. Most slow servers have one binding limit: KV memory, the token budget, the batch width or the CPU of the front end. Know the code path of each knob. Then the start-up log and four metrics show the limit, and you change the right knob.