Troubleshooting vLLM: symptom, cause and fix

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

A vllm serve problem shows up in one of three phases:

1. Start-up: the log never reaches "Application startup complete.". The cause is almost always in the configuration, the memory or the environment.
2. Serving: the server runs, but requests fail (HTTP errors) or are slow. The cause is in the request, the load or a dead engine.
3. Output: HTTP 200, but the content is wrong: tool calls as plain text, rejected JSON schemas, changed quality, a plugin without effect.

The second rule: several processes write to one log (The process and thread architecture of vllm serve). Each line has a prefix, for example (APIServer pid=...) or (EngineCore pid=...). When the engine core fails, the API server only reports that it failed. The real cause is the first traceback in the engine core or worker lines above it.

An everyday comparison. Think of the triage desk of a hospital. The nurse first asks when the problem started and where it hurts, and only then calls the right specialist. The comparison stops at the patient. In vLLM, the "patient" writes its own detailed notes (the tracebacks and log lines). This page tells you which note to read first.

The picture#

Read the lane that matches the phase, from top to bottom. The cards are ordered by how often each problem shows up at start-up or in serving (a judgement, not a measurement). Each card has a step below with the code lines.

In the code#

Step 1: read the right log line#

When the engine core process fails during start-up, it logs the traceback with "EngineCore failed to start." and re-raises (vllm/v1/engine/core.py:1570-1572). A worker process logs "WorkerProc failed to start." (vllm/v1/executor/multiproc_executor.py:957). The API server then raises its own error, which points back to them:

            raise RuntimeError(
                "Engine core initialization failed. "
                "See root cause above. "
                f"Failed core proc(s): {finished}"
                + (
                    f", failed frontend proc(s): {failed_frontend_procs}"
                    if failed_frontend_procs
                    else ""
                )
            )

So scroll up from "Engine core initialization failed. See root cause above." to the first traceback of an EngineCore or Worker line. For more detail, set VLLM_LOGGING_LEVEL=DEBUG (default INFO, vllm/envs.py:860). The vLLM docs also suggest CUDA_LAUNCH_BLOCKING=1 for a failing kernel and NCCL_DEBUG=TRACE for NCCL (docs/usage/troubleshooting.md:33-41).

Step 2: start-up runs out of memory#

Four errors look alike, but they happen at four different points (The KV cache manager: blocks, the block pool and memory sizing and Tuning vLLM: throughput and latency knobs, their real defaults and what each one does explain the memory profile):

--enforce-eager saves the graph memory: the estimate is 0 and no graph is captured (vllm/v1/worker/gpu_worker.py:704-708, vllm/v1/worker/gpu_worker.py:983-985). It costs speed in every step (torch.compile and CUDA graphs in vLLM).

After a successful start, the worker logs the measured parts and a suggested fixed KV size. The text names the flag --kv-cache-memory=N, but the flag is --kv-cache-memory-bytes (vllm/v1/worker/gpu_worker.py:1046-1053, vllm/engine/arg_utils.py:1384-1386). The vLLM docs use the short name too (docs/configuration/optimization.md:24). The short name also works on the command line (derived, not run with vLLM). Python's argparse accepts an unambiguous prefix of a long option, and no vLLM parser sets allow_abbrev=False. FlexibleArgumentParser does not change the prefix matching (vllm/utils/argparse_utils.py:313-502). --kv-cache-memory-bytes is the only flag that starts with --kv-cache-memory.

Step 3: max_model_len is too large#

Two different errors stop the start-up:

1. Larger than the model allows. "User-specified max_model_len (N) is greater than the derived max_model_len (...)" when --max-model-len exceeds the model config. VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 turns this into a warning, with a caution about NaN outputs for RoPE models (vllm/config/model.py:2656-2683).
2. Larger than the KV cache can hold. The message starts "To serve at least one request with the model's max seq len (N), (X GiB KV cache is needed ...". It ends with "... the estimated maximum model length is M." (vllm/v1/core/kv_cache_planning.py:68-87).

For the second error: set --max-model-len M, or --max-model-len auto, which fits the length to the memory and logs "Auto-fit max_model_len: reduced from ..." (vllm/v1/core/kv_cache_planning.py:1584-1587, vllm/v1/core/kv_cache_planning.py:1401-1410). The other fixes are more KV memory (Step 2) or --kv-cache-dtype fp8 (Tuning vLLM: throughput and latency knobs, their real defaults and what each one does). The playground on Tuning vLLM: throughput and latency knobs, their real defaults and what each one does computes the same estimate.

Step 4: start-up is slow#

After the model loads, the engine core profiles, compiles and captures graphs. Then it logs "init engine (profile, create kv cache, warmup model) took X s (compilation: Y s)" (vllm/v1/engine/core.py:394-400). A cold start compiles with torch.compile; a warm start loads the saved result and logs "Directly load AOT compilation from path ..." (vllm/compilation/decorators.py:326-330). The cache lives under VLLM_CACHE_ROOT (vllm/envs.py:713-718; Tour: Start-up of vllm serve).

• Keep the cache on a persistent disk (in a container, a volume). The docs have a section on this for Docker (docs/deployment/docker.md:32).
• A change of max_num_batched_tokens, max_num_seqs or most environment variables gives a new cache key, so the next start compiles again (vllm/config/scheduler.py:271-284).
• For fast restarts during development, use -O0 or --enforce-eager (no compile, no graphs) (vllm/config/vllm.py:1745-1747).

Step 5: "EngineCore failed to start." and the start-up timeouts#

Besides memory and length errors, three causes are common:

• A general plugin raises. The engine core runs every general plugin function, and their errors are not caught (vllm/plugins/__init__.py:87-90). A plugin that fails to import is only logged ("Failed to load plugin", vllm/plugins/__init__.py:68-72). See How vLLM finds and loads plugins.
• A bad environment value. At the end of its __init__, the engine core reads every VLLM_* variable once to cache them (vllm/v1/engine/core.py:241-243, vllm/envs.py:2324-2326). A value that does not parse, for example VLLM_XLA_CHECK_RECOMPILATION=yes (vllm/envs.py:1086-1088), raises there, even when no code uses that variable. With TP > 1, each worker process does the same earlier and logs "WorkerProc failed to start." (vllm/v1/executor/multiproc_executor.py:708-710). A variable that the API server reads when it imports a module fails earlier, in the API server: for example VLLM_ENGINE_READY_TIMEOUT_S (vllm/v1/engine/core_client.py:29).
• A wrong flag combination that only the engine core checks, for example an attention backend that the GPU does not support (How vLLM chooses an attention backend).

Timeouts at start-up depend on the shape:

• With one API server, the front end waits for the engine with no time limit (The engine client: AsyncLLM in the API server).
• With more than one API server, the API servers start before the engines and wait VLLM_ENGINE_READY_TIMEOUT_S (default 600 s) (vllm/envs.py:812-814, vllm/v1/engine/core_client.py:755-768). The message says that slow weight loading is a common cause; raise the value for large models.
• The engine core waits at most 5 minutes for the init message of the front end (vllm/v1/engine/core.py:113, vllm/v1/engine/core.py:1408-1413).

Step 6: the engine dies while serving#

An error in the loop of the engine core makes it log "EngineCore encountered a fatal error.". Then it sends a dead message to the front end and exits (vllm/v1/engine/core.py:1573-1576). The front end then raises EngineDeadError (vllm/v1/engine/core_client.py:581-584). The result is engine death:

• Requests get HTTP 500 with the message "The server had an error while processing your request." (vllm/v1/engine/exceptions.py:12-22, vllm/entrypoints/serve/exception_handling/error_response.py:68-72).
• GET /health answers 503 (vllm/entrypoints/serve/instrumentator/health.py:29-33).
• A watchdog checks every 5 s and stops the API server, unless VLLM_KEEP_ALIVE_ON_ENGINE_DEATH=1 (default 0) (vllm/entrypoints/launchers/launcher.py:314-337, vllm/envs.py:1107-1109). Let your orchestrator restart the container.

With worker processes, each execute_model call waits at most VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS (default 300) for the workers (vllm/envs.py:1859-1863, vllm/v1/executor/multiproc_executor.py:350). Then it raises TimeoutError("RPC call to execute_model timed out.") (vllm/v1/executor/multiproc_executor.py:443). This is an error in the engine core loop, so the engine dies as above (derived; with fault tolerance off, the default). The comment in envs.py says "only applies when TP > 1"; the code applies it to every multiprocessing executor, also with PP or DP. A hang inside a collective (NCCL) usually shows up as this timeout (Step 10).

Step 7: HTTP 400 and 404#

The error body names the cause and often the field (The OpenAI-compatible API). It has the form {"error": {"message": "...", "type": "BadRequestError", "param": "...", "code": 400}}. The common ones:

A max_tokens larger than the rest of the context is not an error: the server lowers it to max_model_len − prompt length (vllm/entrypoints/serve/utils/api_utils.py:253-270). For a 404, check GET /v1/models: the served name is --served-model-name, else the model path (vllm/config/model.py:2330-2334).

Step 8: slow serving: preemption and prefix-cache misses#

Preemption. When running requests need more blocks than are free, the scheduler preempts one and recomputes it later (The V1 scheduler: one token budget, no phases). The scheduler writes no log line per preemption. The signs are the counter vllm:num_preemptions_total and "Preemptions: N" in the periodic log line (vllm/v1/metrics/loggers.py:738-745, vllm/v1/metrics/loggers.py:310-312). Watch the preemption rate: rate(vllm:num_preemptions_total[5m]). The vLLM docs still show a V0-style warning line for preemption (docs/configuration/optimization.md:33-35); the current code does not print it. Fix: more KV memory, or fewer running requests (Tuning vLLM: throughput and latency knobs, their real defaults and what each one does).

A low prefix-cache hit rate. The cache hits only when the hash chain of full blocks matches (Prefix caching in the engine core: hashes, lookup, caching, eviction and KV events). Hits break when:

• the shared prefix is shorter than one block, or the difference starts inside the first block: only full blocks hit (vllm/v1/core/kv_cache_manager.py:264-266);
• requests use different LoRA adapters, multimodal inputs or cache_salt values, which are part of the block hash (vllm/v1/core/kv_cache_utils.py:655-665);
• a request asks for prompt_logprobs: it skips reading the cache (vllm/sampling_params.py:625-632);
• the chat template puts a changing value (a date, an id) near the start of the prompt (general knowledge about templates);
• you use a non-cryptographic hash (xxhash) without PYTHONHASHSEED. Then the hash seed is random per process, and tools that compare hashes across instances find no match (vllm/v1/core/kv_cache_utils.py:117-130).

Also note that requests that come back after a preemption are not in the exported prefix counters (Metrics and observability: how to watch a vLLM server).

Step 9: wrong output with HTTP 200#

• Tool calls arrive as plain text in content. The server needs both --enable-auto-tool-choice and --tool-call-parser NAME. The first alone stops the start-up with TypeError (vllm/entrypoints/common/cli_args.py:523-525); the second alone gives no parser (vllm/parser/parser_manager.py:34-35). The parser must match the model's output format (Tool and reasoning parsers: from model text to tool_calls and reasoning).
• Structured output requests fail after other ones worked. The API server picks a backend per request, but the engine core keeps the first backend it built and rejects others (vllm/v1/structured_output/__init__.py:143-150). Set one backend with --structured-outputs-config.backend (Structured outputs: JSON schemas, grammars and token bitmasks in vLLM).
• The output quality changed. By default, the server takes default sampling parameters from the model's generation_config.json; --generation-config vllm uses vLLM's own defaults (vllm/config/model.py:358-364). The vLLM docs describe this change (docs/usage/troubleshooting.md:25-31).
• A plugin does not load. When VLLM_PLUGINS is set, only the named plugins load; an empty value loads none (vllm/envs.py:1161-1167, vllm/plugins/__init__.py:63-66). General plugins log at DEBUG level only, so set VLLM_LOGGING_LEVEL=DEBUG to see "Available plugins for group ..." (vllm/plugins/__init__.py:48-54). Logits-processor plugins ignore VLLM_PLUGINS (How vLLM finds and loads plugins).
• A custom op switch has no effect. custom_ops entries use the registered op name, for example +rms_norm. A name that matches no op of the model only logs a WARNING "... has no effect" (vllm/config/compilation.py:1343-1361). Also, at the default -O2 with Inductor, vLLM disables most custom ops on purpose (CustomOp, out-of-tree ops and IR op providers).

Step 10: multi-node and NCCL hangs#

The vLLM docs have the details; this page points to them:

• Check GPU-to-GPU communication across the nodes first, and set network variables such as NCCL_SOCKET_IFNAME on all nodes (docs/serving/distributed_troubleshooting.md:5-7).
• With several IP addresses per node, set VLLM_HOST_IP per node (docs/serving/distributed_troubleshooting.md:9-11).
• A test script with NCCL_DEBUG=TRACE shows whether the hardware and drivers work (docs/usage/troubleshooting.md:83-175).
• vLLM sets a few variables itself when it is imported: PYTORCH_NVML_BASED_CUDA_CHECK=1 and TORCHINDUCTOR_COMPILE_THREADS=1 (vllm/env_override.py:164-168). It sets NCCL_GRAPH_REGISTER=0 only when sleep mode offloads CUDA graphs (vllm/config/vllm.py:2309-2316). Older versions set NCCL_CUMEM_ENABLE=0; the current code does not (docs/usage/troubleshooting.md:375).
• Traffic between nodes is not encrypted; keep the nodes on an isolated network (docs/usage/security.md:3-10).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

What can go wrong#

Troubleshooting itself has traps:

• Reading the last error instead of the first. The API server's "Engine core initialization failed" is a consequence; the cause is above it (Step 1).
• Copying the suggested --kv-cache-memory=N to another machine. The value is valid only for the same GPU with the same free memory (Step 2).
• Trusting old docs for preemption. The V0 warning line in the docs does not exist in the current code (Step 8).
• Raising VLLM_ENGINE_READY_TIMEOUT_S with one API server. It has no effect there (Step 5).
• Setting VLLM_KEEP_ALIVE_ON_ENGINE_DEATH=1 in production. The server stays up but answers 500 to every request; the orchestrator does not restart it unless it checks /health (Step 6).

Why it matters. Most vLLM problems leave a precise message in the right process log. Know the phase, the process and the line of code that wrote the message. Then a fix usually takes minutes instead of hours.