Metrics and observability: how to watch a vLLM server

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

A running vllm serve has four places where you can see what it does. All four get their numbers from the same source. The EngineCore sends events, timestamps and counters with its outputs, and the front end computes the metrics from them. Serving metrics: TTFT, TPOT, ITL, throughput, goodput defines TTFT, ITL, TPOT and E2E latency.

• Prometheus: GET /metrics on the API server port, with about 130 vllm:* metrics (Prometheus metrics lists them all).
• The log line: every 10 seconds, "Avg prompt throughput ... Running ... Waiting ... GPU KV cache usage ...".
• Per request: an optional metrics object in each response, and an optional OpenTelemetry span per request.
• Profiles: the torch profiler, started and stopped with two HTTP routes.

This page shows where each signal is made, what it really measures, and which flag turns it on or off. It ends with the known gaps, because several metrics do not mean what their names suggest.

An everyday comparison. Think of a car. The engine sends raw pulses from its sensors; the dashboard computes the speed and the fuel use from them. A trip computer can also print one summary every few minutes. The comparison stops at the clock. In vLLM, the dashboard (the API server) and the engine (the engine core) use two different clocks. Some times come from the front-end clock, others from the engine clock.

The picture#

Read the panels from top to bottom. Green boxes run in the engine core process, blue boxes in the API server. The engine core never computes a latency. It sends time.monotonic() timestamps, and the API server adds its own time.time() clock for TTFT and E2E latency.

In the code#

Step 1: the /metrics endpoint#

The API server mounts the endpoint when it registers its routes (vllm/entrypoints/serve/instrumentator/__init__.py:16-18):

def attach_router(app: FastAPI):
    """Mount prometheus metrics to a FastAPI app."""
    registry = get_prometheus_registry()

    # `response_class=PrometheusResponse` is needed to return an HTTP response
    # with header "Content-Type: text/plain; version=0.0.4; charset=utf-8"
    # instead of the default "application/json" which is incorrect.
    # See https://github.com/trallnag/prometheus-fastapi-instrumentator/issues/163#issue-1296092364
    Instrumentator(
        excluded_handlers=[
            "/metrics",
            "/health",
            "/load",
            "/ping",
            "/version",
            "/server_info",
        ],
        registry=registry,
    ).add().instrument(app).expose(app, response_class=PrometheusResponse)

    # Add prometheus asgi middleware to route /metrics requests
    metrics_route = Mount("/metrics", make_asgi_app(registry=registry))

    # Workaround for 307 Redirect for /metrics
    metrics_route.path_regex = re.compile("^/metrics(?P<path>.*)$")
    app.routes.append(metrics_route)

1. Lines 64-74: the external library prometheus_fastapi_instrumentator adds http_* metrics for every route except the six excluded ones. Its expose() call adds a /metrics route.
2. Line 77: a second route mounts the prometheus_client ASGI app at /metrics; line 80 makes it also match /metrics/ and longer paths. Both routes read the same registry (line 58).

With one API server, the registry is the default prometheus_client registry (vllm/v1/metrics/prometheus.py:47-53). With --api-server-count above 1, each API server is its own process. Then vllm serve sets PROMETHEUS_MULTIPROC_DIR to a new temporary folder before it starts them (vllm/entrypoints/cli/serve.py:282-283, vllm/v1/metrics/prometheus.py:17-36). Each process writes its values there, and every /metrics answer merges all processes. If you set the variable yourself, vLLM warns that you must clean the folder between runs (vllm/v1/metrics/prometheus.py:30-36).

Almost every engine metric has the labels model_name (the first served name) and engine (the DP rank as text, for example "0") (vllm/v1/metrics/loggers.py:499-504). The label rules are on Prometheus metrics.

Step 2: the metrics to watch first#

The table lists the exported names, which are the names you query. prometheus_client adds _total to a counter and _bucket, _sum and _count to a histogram.

Step 3: PromQL recipes#

These queries use only the exported names above. Add model_name or engine to the by clauses when you serve several models or DP ranks.

# p99 TTFT over 5 minutes, all engines together
histogram_quantile(0.99, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))

# p50 ITL per engine
histogram_quantile(0.5, sum by (le, engine) (rate(vllm:inter_token_latency_seconds_bucket[5m])))

# mean E2E latency
rate(vllm:e2e_request_latency_seconds_sum[5m]) / rate(vllm:e2e_request_latency_seconds_count[5m])

# output tokens per second
sum(rate(vllm:generation_tokens_total[1m]))

# prefix cache hit rate (tokens)
rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m])

# preemptions per second
sum(rate(vllm:num_preemptions_total[5m]))

# KV cache usage in percent (the gauge is 0-1)
100 * vllm:kv_cache_usage_perc

# finished requests per second by reason
sum by (finished_reason) (rate(vllm:request_success_total[5m]))

Step 4: the periodic log line#

When stats are on, the API server starts a task that calls do_log_stats() every VLLM_LOG_STATS_INTERVAL seconds (vllm/entrypoints/launchers/utils/server_utils.py:75-89). The default is 10.0; a value of 0 or below becomes 10.0 (vllm/envs.py:874-878). The LoggingStatLogger builds the line:

    def log(self):
        self._update_stats()
        self.aggregate_scheduler_stats()
        # Avoid log noise on an idle production system
        log_fn = logger.debug if self.engine_is_idle else logger.info
        # Format and print output.
        log_parts = [
            "Avg prompt throughput: %.1f tokens/s",
            "Avg generation throughput: %.1f tokens/s",
            "Running: %d reqs",
            "Waiting: %d reqs",
        ]
        total_waiting = (
            self.last_scheduler_stats.num_waiting_reqs
            + self.last_scheduler_stats.num_skipped_waiting_reqs
        )
        log_args: list[int | float | str] = [
            self.last_prompt_throughput,
            self.last_generation_throughput,
            self.last_scheduler_stats.num_running_reqs,
            total_waiting,
        ]

        if self.last_scheduler_stats.num_skipped_waiting_reqs > 0:
            log_parts.append("Deferred: %d reqs")
            log_args.append(self.last_scheduler_stats.num_skipped_waiting_reqs)

        kv_fetch = self.last_scheduler_stats.num_kv_fetch_reqs_by_stage
        if any(kv_fetch.values()):
            log_parts.append(
                "KV fetch: %d waiting to start, %d in progress, %d completed waiting"
            )
            log_args.extend(kv_fetch.get(stage, 0) for stage in KV_FETCH_STAGES)

        if self.num_preemptions > 0:
            log_parts.append("Preemptions: %d")
            log_args.append(self.num_preemptions)

1. Line 280: an idle server logs at DEBUG, so the line disappears at the default INFO level. Idle means no prompt or output tokens in the last two intervals (vllm/v1/metrics/loggers.py:255-265).
2. Lines 283-286: the two throughputs are tokens since the last line divided by the elapsed time (vllm/v1/metrics/loggers.py:168-173). The prompt throughput counts computed prompt tokens, not prefix-cache hits (vllm/v1/metrics/loggers.py:162-163).
3. Lines 288-301: "Waiting" includes deferred requests, and "Deferred" appears only when there are some.
4. Lines 310-312: "Preemptions: N" appears only when there were preemptions in the interval. The scheduler itself logs nothing when it preempts (The V1 scheduler: one token budget, no phases).

The line continues with "GPU KV cache usage: x%" and, after the first lookup, "Prefix cache hit rate: y%" (vllm/v1/metrics/loggers.py:314-323). The hit rate here covers about the last 1,000 requests, not the whole run (vllm/v1/metrics/stats.py:47). Each line starts with the engine index, for example "Engine 000: ", also with one engine (vllm/v1/metrics/loggers.py:176-178, vllm/v1/metrics/loggers.py:336). With several engines behind one API server, each engine gets its own line, unless --aggregate-engine-logging gives one line "N Engines Aggregated: " (vllm/v1/metrics/loggers.py:371-373, vllm/v1/metrics/loggers.py:1390-1395). With speculative decoding, a second line "SpecDecoding metrics: ..." follows (vllm/v1/metrics/loggers.py:340).

Step 5: per-request metrics in the response#

--enable-per-request-metrics (off by default, vllm/entrypoints/common/cli_args.py:150-151) adds a metrics object to each chat or completion response. It needs the stats: with --disable-log-stats, start-up stops with ValueError (vllm/entrypoints/common/cli_args.py:536-542). The fields come from the engine-core timestamps of the request:

    if scheduled_ts > 0 and first_token_ts > 0:
        time_to_first_token_ms = (first_token_ts - scheduled_ts) * 1000

    if first_token_ts > 0 and last_token_ts > 0:
        generation_time_ms = (last_token_ts - first_token_ts) * 1000

    if queued_ts > 0 and scheduled_ts > 0:
        queue_time_ms = (scheduled_ts - queued_ts) * 1000

    if first_token_ts > 0 and last_token_ts > 0 and num_generation_tokens > 1:
        decode_time = last_token_ts - first_token_ts
        mean_itl_ms = decode_time / (num_generation_tokens - 1) * 1000

    if scheduled_ts > 0 and last_token_ts > 0:
        inference_time_ms = (last_token_ts - scheduled_ts) * 1000
        if inference_time_ms > 0:
            tokens_per_second = num_generation_tokens / inference_time_ms * 1000

Note line 74: this time_to_first_token_ms runs from the first SCHEDULED event to the first token. It is the prefill time on the engine clock, without the queue and without the time in the API server. It is shorter than the Prometheus TTFT of the same request, because its interval lies inside the Prometheus interval (derived). In a stream, the object comes only in the final usage chunk, so the client must ask for usage (stream_options.include_usage) (vllm/entrypoints/openai/chat_completion/serving.py:873-885). For n > 1, the chat server sends no metrics, because the stats belong to one of the n outputs only (vllm/entrypoints/openai/chat_completion/serving.py:1173-1176).

Step 6: OpenTelemetry traces#

--otlp-traces-endpoint URL turns tracing on (vllm/config/observability.py:38-39). The value is checked at start-up: without the OpenTelemetry packages, it raises ValueError (vllm/config/observability.py:153-165). AsyncLLM then sets up a tracer (vllm/v1/engine/async_llm.py:149-151). The exporter uses gRPC by default; OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf switches to HTTP (vllm/tracing/otel.py:94-101).

When a request finishes, the output processor makes one span named llm_request. It starts at arrival_time and carries the queue, prefill, decode and inference times, the token counts and some sampling parameters (vllm/v1/engine/output_processor.py:957-1001). The serving classes pass the traceparent and tracestate headers of the HTTP request, so the span joins the caller's trace (vllm/tracing/utils.py:12, vllm/entrypoints/openai/chat_completion/serving.py:378-382). A request with trace headers to a server without tracing logs one warning and drops the headers (vllm/v1/engine/async_llm.py:421-423). --collect-detailed-traces model,worker adds model forward and execute times, with a cost in speed (vllm/config/observability.py:41-48).

Step 7: stat logger plugins#

A plugin in the entry-point group vllm.stat_logger_plugins adds its own stat logger (vllm/v1/metrics/loggers.py:84-98; Plugins in the API server). AsyncLLM loads the plugins in every API server process. When a plugin is found, stats stay on even with --disable-log-stats; only the default log-line logger is left out (vllm/v1/engine/async_llm.py:154-164). The manager adds a PrometheusStatLogger unless a custom logger already is one (vllm/v1/metrics/loggers.py:1396-1419). The plugin's record() runs once per batch of engine outputs, and log() every log interval.

Step 8: the profiler routes#

POST /start_profile and POST /stop_profile exist only when --profiler-config names a profiler; the server then logs that this "should ONLY be used for local development" (vllm/entrypoints/serve/profile/api_router.py:56-65). The profilers are torch, cuda and proton (vllm/config/profiler.py:15, vllm/config/profiler.py:70-76). The torch profiler needs torch_profiler_dir (vllm/config/profiler.py:235-241). A start request can pass profile_prefix, delay_iterations and max_iterations (vllm/entrypoints/serve/profile/api_router.py:18-28). An example (not in the repository):

vllm serve Qwen/Qwen3-0.6B --profiler-config '{"profiler": "torch", "torch_profiler_dir": "/tmp/vllm_profile"}'
curl -X POST localhost:8000/start_profile -H "Content-Type: application/json" -d '{"max_iterations": 20}'
# send some requests, then:
curl -X POST localhost:8000/stop_profile

Step 9: what --disable-log-stats turns off#

--disable-log-stats sets log_stats to False (vllm/engine/arg_utils.py:1909-1914). Without a stat logger plugin, AsyncLLM then makes no stat logger manager (vllm/v1/engine/async_llm.py:203-213), and:

• /metrics has no vllm:* engine metrics; the http_* metrics stay (Prometheus metrics);
• the log line stops, because the log task starts only with stats on (vllm/entrypoints/launchers/utils/server_utils.py:78);
• the engine core records no QUEUED, SCHEDULED or PREEMPTED events (vllm/v1/core/sched/scheduler.py:2746-2747);
• requests carry no stats, so per-request metrics and traces have no data.

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

(APIServer pid=4121) INFO ... Engine 000: Avg prompt throughput: 5120.4 tokens/s, Avg generation throughput: 1835.0 tokens/s, Running: 64 reqs, Waiting: 12 reqs, Preemptions: 3, GPU KV cache usage: 97.8%, Prefix cache hit rate: 41.2%

What can go wrong#

• /metrics has only http_* metrics. The server runs with --disable-log-stats (Step 9).
• No log line. More than one API server (Step 4), an idle server (DEBUG level), or VLLM_LOGGING_LEVEL above INFO.
• A dashboard shows KV usage of 0.98 %. The gauge is a fraction; multiply by 100 (vllm/v1/metrics/loggers.py:621).
• Old metrics files after a restart. You set PROMETHEUS_MULTIPROC_DIR yourself and did not clean it (vllm/v1/metrics/prometheus.py:30-36).
• Per-request TTFT disagrees with the dashboard. They measure different intervals (Step 5).
• /start_profile answers 404. No --profiler-config with a profiler (Step 8).
• The client and the server disagree on TTFT. The client clock starts earlier and stops later; see Benchmarking vLLM: vllm bench serve, throughput, latency and startup.

Why it matters. You can tune and debug a server only through these signals. Each one has a start point, a clock and a switch. If you know them, you can compare the right numbers and find the binding limit (Troubleshooting vLLM: symptom, cause and fix).