Prometheus metrics

Code version: vLLM @ 3e44c0d · 2026-10-22

Use this page in two ways. To learn the mechanism, read the first four sections. To look up one metric, go to All metrics, or search for its name. The script tools/gen_metrics.py makes the tables from the vLLM source. The text above the tables is written by hand from the code. For what TTFT, ITL and TPOT mean, read Serving metrics: TTFT, TPOT, ITL, throughput, goodput first.

Where /metrics is served#

The API server mounts the endpoint when it registers its routes (vllm/entrypoints/serve/instrumentator/__init__.py:16-18). attach_router does two things:

def attach_router(app: FastAPI):
    """Mount prometheus metrics to a FastAPI app."""
    registry = get_prometheus_registry()

    # `response_class=PrometheusResponse` is needed to return an HTTP response
    # with header "Content-Type: text/plain; version=0.0.4; charset=utf-8"
    # instead of the default "application/json" which is incorrect.
    # See https://github.com/trallnag/prometheus-fastapi-instrumentator/issues/163#issue-1296092364
    Instrumentator(
        excluded_handlers=[
            "/metrics",
            "/health",
            "/load",
            "/ping",
            "/version",
            "/server_info",
        ],
        registry=registry,
    ).add().instrument(app).expose(app, response_class=PrometheusResponse)

    # Add prometheus asgi middleware to route /metrics requests
    metrics_route = Mount("/metrics", make_asgi_app(registry=registry))

    # Workaround for 307 Redirect for /metrics
    metrics_route.path_regex = re.compile("^/metrics(?P<path>.*)$")
    app.routes.append(metrics_route)

1. Lines 64-74 set up the external library prometheus_fastapi_instrumentator with the registry from get_prometheus_registry(). Its HTTP middleware records http_* metrics for each route, except /metrics, /health and four other routes. Its expose() call also adds a /metrics route of that library.
2. Line 77 mounts the prometheus_client ASGI app at /metrics, with the same registry. Line 80 lets this mount also match paths below /metrics.

With one API server, the registry is the default prometheus_client registry. With --api-server-count above 1, vllm serve first sets PROMETHEUS_MULTIPROC_DIR to a new temporary folder, if you did not set it (vllm/entrypoints/cli/serve.py:282-283, vllm/v1/metrics/prometheus.py:21-26). Then each API server process writes its values into that folder, and /metrics merges them with a MultiProcessCollector (vllm/v1/metrics/prometheus.py:47-53). For this reason, the gauges declare a multiprocess_mode, for example "mostrecent" (vllm/v1/metrics/loggers.py:526-531).

Which process records the metrics#

The engine core sends its statistics to the front end with its outputs. The front end turns them into Prometheus values:

1. When stats are on, AsyncLLM makes a StatLoggerManager (vllm/v1/engine/async_llm.py:205-213). The manager always adds a PrometheusStatLogger, unless a custom stat logger is already one (vllm/v1/metrics/loggers.py:1406-1407, vllm/v1/metrics/loggers.py:1416-1419).
2. The output handler of AsyncLLM gets each batch of EngineCoreOutputs and makes an IterationStats for it (vllm/v1/engine/async_llm.py:880-885). IterationStats takes the front-end clock time.time() when it is made (vllm/v1/metrics/stats.py:477-478).
3. The output processor fills it. TTFT is that time minus the arrival time of the request (vllm/v1/metrics/stats.py:525-527). ITL is the gap between two engine-core timestamps (vllm/v1/metrics/stats.py:553-558).
4. The handler calls record() with the scheduler stats of the engine core and the iteration stats (vllm/v1/engine/async_llm.py:929-935). PrometheusStatLogger.record sets the gauges, increments the counters and observes the histograms (vllm/v1/metrics/loggers.py:1138-1166, vllm/v1/metrics/loggers.py:1284-1287).

With more than one API server, the log-line stat logger is off ("to avoid incomplete stats"), but the Prometheus logger stays on (vllm/v1/metrics/loggers.py:1385-1389, vllm/v1/metrics/loggers.py:1416-1419).

Labels: model_name and engine#

Almost every engine metric has two labels (vllm/v1/metrics/loggers.py:499-504):

• model_name is served_model_name. If you give several names with --served-model-name, the label gets the first one. If you give none, it gets the model (vllm/config/model.py:2330-2334, vllm/config/model.py:600-603).
• engine is the index of the engine core as text, for example "0". With data parallelism serving, each API server records one series per engine core that it manages (vllm/v1/engine/async_llm.py:208).

Some metrics add a third label, for example finished_reason on vllm:request_success. The generated table shows all labels. In the helper classes, for example the speculative decoding and KV connector metrics, a † marks the two labels that the stat logger passes in.

Metrics to put on your dashboard first#

This list is a choice for online serving. The "Exposed as" names are what you query: prometheus_client adds _total to a counter and _bucket, _sum and _count to a histogram.

Known issues#

--show-hidden-metrics-for-version has no effect#

ObservabilityConfig.show_hidden_metrics is True only when --show-hidden-metrics-for-version names the previous minor version (vllm/config/observability.py:31-36). PrometheusStatLogger stores the value (vllm/v1/metrics/loggers.py:494), but no code in vllm/ reads self.show_hidden_metrics after that. So at this commit, the flag does not show or hide any metric.

The docs describe a different behaviour: a metric deprecated in X.Y is hidden in X.Y+1 and comes back with --show-hidden-metrics-for-version=X.Y (docs/usage/metrics.md:132-136). The code has no hidden metric now. This is a difference between docs and code.

Three deprecated KV offload metrics#

The offloading connector still declares vllm:kv_offload_total_bytes, vllm:kv_offload_total_time and vllm:kv_offload_size as "deprecated legacy transfer metrics" with a transfer_type label (vllm/distributed/kv_transfer/kv_connector/v1/offloading/metrics.py:149-167). It registers them with every offloading connector (vllm/distributed/kv_transfer/kv_connector/v1/offloading/metrics.py:409-433), but observes them only when the offloading spec is a CPUOffloadingSpec (vllm/distributed/kv_transfer/kv_connector/v1/offloading/metrics.py:402, vllm/distributed/kv_transfer/kv_connector/v1/offloading/metrics.py:505-508). Use the new flat metrics of the same connector in the table.

Four KVCR metrics come from outside the repository#

The KVCR tier of the KV offload code names four metrics with constants from the package kvcr, which is not in the repository (vllm/v1/kv_offload/tiering/kvcr/manager.py:15-21). The code adds the prefix vllm: (vllm/v1/kv_offload/tiering/kvcr/manager.py:120-121, vllm/v1/kv_offload/tiering/kvcr/manager.py:124-142). The scan cannot resolve the names, so the table lists only the four source lines.

HTTP and process metrics are not in the table#

The http_* metrics come from prometheus_fastapi_instrumentator (see above). The docs show an example: http_requests_total, http_request_size_bytes, http_response_size_bytes, http_request_duration_highr_seconds and http_request_duration_seconds (docs/design/metrics.md:89-96). The process_* and python_* metrics, if present, come from prometheus_client. Its default registry has these collectors; the multiprocess registry that vllm serve uses with more than one API server does not (vllm/v1/metrics/prometheus.py:47-53). vLLM code does not declare these metrics, so the generated table does not list them.

How to read the tables#

• Groups follow the file that declares the metric, for example "Engine core (PrometheusStatLogger)" for vllm/v1/metrics/loggers.py. The groups are the tool's choice.
• Type and labels: counter, gauge or histogram, the label names, and the buckets and bucket family of a histogram. In the tables of the helper classes, a † marks the labels that come from the labelnames parameter. The PrometheusStatLogger metrics have the same two labels without a †.
• Documentation string is the documentation= text of the metric, quoted as it is. Created only if gives the if conditions around the declaration. Such a metric exists only when the condition is true, for example vllm:lora_requests_info needs a LoRA config.
• Declared links the line of the name and the constructor call.

All metrics#

128 metric names that start with vllm:, declared in vLLM code: 62 counters, 25 gauges, 41 histograms. The stat loggers use the labels model_name and engine (v1/metrics/loggers.py:499). A † marks these two labels only where a helper class (speculative decoding, performance, KV connectors) gets them through its labelnames parameter. Most PrometheusStatLogger metrics have the same two labels, without a †. 3 metrics are marked deprecated; no metric checks show_hidden_metrics at this commit.

Not in the table: 4 names this scan cannot resolve (they come from code outside the repo): _vllm_metric_name(DURATION_METRIC) (v1/kv_offload/tiering/kvcr/manager.py:126); _vllm_metric_name(TRANSFER_BYTES_METRIC) (v1/kv_offload/tiering/kvcr/manager.py:130); _vllm_metric_name(TRANSFER_BLOCKS_METRIC) (v1/kv_offload/tiering/kvcr/manager.py:134); _vllm_metric_name(STATE_METRIC) (v1/kv_offload/tiering/kvcr/manager.py:138).

Engine core (PrometheusStatLogger) (43)#

Performance estimates (MFU) (3)#

Speculative decoding and diffusion (7)#

KV offloading connector (37)#

Simple KV offload (5)#

KV connector: NIXL (8)#

KV connector: Mooncake store (5)#

KV connector: 3FS (4)#

KV connector: HiSparse (8)#

API server (8)#