Serving metrics: TTFT, TPOT, ITL, throughput, goodput

vLLM computes an interval only from two timestamps of the same process. The design document gives the reason: monotonic clocks of two processes have different start points (docs/design/metrics.md:177-182).

So TTFT and E2E latency are front-end intervals: iteration_timestamp − arrival_time (vllm/v1/metrics/stats.py:505-507). They include the chat template, tokenization, the trip to the engine core and back, and the wait. They do not include HTTP parsing before rendering, detokenization after the output arrives, or the network to the client. Queue, prefill, decode, ITL and TPOT are engine-core intervals.

The design document says: "Currently arrival_time starts when tokenization begins." (docs/design/metrics.md:262-263). In the code, it starts a little earlier: before the chat template (render_messages_async, vllm/renderers/base.py:1385-1388).

Chunked prefill. The scheduler sends an output for a request only when it has new tokens, a pooling output or a stop (vllm/v1/core/sched/scheduler.py:2293-2295). A prompt chunk that makes no token sends nothing. So the first output, and the end of TTFT, comes after the last chunk of the prompt.

When a request finishes, the output processor calls update_from_finished_request() (vllm/v1/engine/output_processor.py:885-887):

    def update_from_finished_request(
        self,
        finish_reason: "FinishReason",
        request_id: str,
        num_prompt_tokens: int,
        max_tokens_param: int | None,
        req_stats: RequestStateStats,
        num_cached_tokens: int = 0,
    ):
        e2e_latency = self._time_since(req_stats.arrival_time)

        # Queued interval is from first QUEUED event to first SCHEDULED
        queued_time = req_stats.scheduled_ts - req_stats.queued_ts

        # Prefill interval is from first SCHEDULED to first NEW_TOKEN
        # Any preemptions during prefill is included in the interval
        prefill_time = req_stats.first_token_ts - req_stats.scheduled_ts

        # Decode interval is from first NEW_TOKEN to last NEW_TOKEN
        # Any preemptions during decode are included
        decode_time = req_stats.last_token_ts - req_stats.first_token_ts

        # Inference interval is from first SCHEDULED to last NEW_TOKEN
        # Any preemptions during prefill or decode are included
        inference_time = req_stats.last_token_ts - req_stats.scheduled_ts

        # Do not count the token generated by the prefill phase
        mean_time_per_output_token = (
            decode_time / (req_stats.num_generation_tokens - 1)
            if req_stats.num_generation_tokens - 1 > 0
            else 0
        )

1. Line 594. E2E latency = front-end time now − arrival_time.
2. Line 597. Queue time = first SCHEDULED − QUEUED.
3. Line 601. Prefill time = first token − first SCHEDULED. A preemption during prefill stays inside this interval (comment on line 600).
4. Line 605. Decode time = last token − first token.
5. Line 609. Inference time = last token − first SCHEDULED.
6. Lines 612-616. TPOT = decode time / (generated tokens − 1). The first token comes from prefill, so it is not counted. A request with 0 or 1 output token gets TPOT 0.

Docs and code differ on TPOT and ITL. The design document gives TPOT as (E2E latency − TTFT) / (output tokens − 1) (docs/design/metrics.md:55). The code divides the decode interval instead (lines 605 and 612-616). The document also calls each ITL sample a "wall-clock gap" between streamed outputs (docs/design/metrics.md:69-70). The code uses the monotonic timestamps of two engine outputs (vllm/v1/metrics/stats.py:554-556).

Why "first" SCHEDULED. A preempted request is scheduled again later. The code keeps only the first value: if req_stats.scheduled_ts == 0.0: # ignore preemptions (vllm/v1/metrics/stats.py:576-578). The design document instead says "between QUEUED and most recent SCHEDULED" (docs/design/metrics.md:212). The code wins: queue time does not grow after a preemption, but prefill or decode time does.

A preemption frees all blocks of the request and sets num_computed_tokens to 0. The request goes back to the front of the waiting queue (vllm/v1/core/sched/scheduler.py:1676-1700). It records a PREEMPTED event, which counts in vllm:num_preemptions (vllm/v1/metrics/stats.py:580-583).

Worked example (the request of the diagram). arrival_time = 0, queued_ts = 5 ms, scheduled_ts = 55 ms, first token = 200 ms, last token = 2,200 ms, 101 output tokens. Then queue = 50 ms, prefill = 145 ms and TTFT = 200 ms. Decode = 2,000 ms, inference = 2,145 ms, TPOT = 2,000 / 100 = 20 ms and E2E = 2,200 ms.

The API server serves Prometheus metrics at /metrics (vllm/entrypoints/serve/instrumentator/metrics.py:77). A Prometheus histogram counts samples in buckets. Each name gets the series _bucket, _sum and _count.

The bucket families are in one file (vllm/v1/metrics/loggers.py:831-839). The TTFT buckets go from 1 ms to 2,560 s (vllm/v1/metrics/buckets.py:57-80). The ITL buckets go from 10 ms to 80 s (vllm/v1/metrics/buckets.py:84-104). The request-latency buckets go from 0.3 s to 7,680 s (vllm/v1/metrics/buckets.py:30-52). You can replace a family with the config field custom_histogram_buckets (vllm/config/observability.py:70, vllm/v1/metrics/buckets.py:232-233).

Why buckets matter. Prometheus estimates a percentile from the bucket counts. With the default ITL buckets, all gaps between 10 ms and 25 ms fall into one bucket. So a p99 ITL from Prometheus is only as exact as the bucket edges.

Throughput counters and the log line. The counters vllm:prompt_tokens and vllm:generation_tokens grow by the tokens of each output (vllm/v1/metrics/loggers.py:1262-1272). The LoggingStatLogger also writes a log line with "Avg prompt throughput" and "Avg generation throughput" (vllm/v1/metrics/loggers.py:283-284). It divides the tokens by the time since the last log (vllm/v1/metrics/loggers.py:169-174). The API server writes this line every VLLM_LOG_STATS_INTERVAL seconds, default 10 (vllm/envs.py:874-878, vllm/entrypoints/launchers/utils/server_utils.py:79-84).

Turn off. The API server gives --disable-log-stats to AsyncLLM (vllm/entrypoints/launchers/api_server/entry.py:99). Then AsyncLLM makes no default stat loggers. A stat-logger plugin still turns statistics on, but only for the plugin loggers (vllm/v1/engine/async_llm.py:158-159, vllm/v1/engine/async_llm.py:204-210).

vllm bench serve is a subcommand that calls vllm/benchmarks/serve.py (vllm/entrypoints/cli/benchmark/serve.py:10-21). It sends requests to a running server and times them at the client.

One request. For the chat API, the client sends "stream": true with usage in the last chunk (vllm/benchmarks/lib/endpoint_request_func.py:423-432). It starts a time.perf_counter() clock before the POST (vllm/benchmarks/lib/endpoint_request_func.py:443-445). Then:

1. TTFT = the time of the first SSE chunk with output − start (line 477).
2. Each later chunk with output adds one ITL sample: the gap to the previous chunk with output (lines 481, 486).
3. The usage chunk gives the output token count, but it does not move the end time (lines 484-490).
4. Latency (E2EL) = the time of the last output chunk − start (vllm/benchmarks/lib/endpoint_request_func.py:501).

All requests. calculate_metrics() computes (vllm/benchmarks/serve.py:654-660, vllm/benchmarks/serve.py:778-781):

• TPOT per request = (latency − TTFT) / (output tokens − 1). Requests with 1 output token are not in the TPOT statistics.
• Request throughput = completed / duration. Output token throughput = output tokens / duration. Total token throughput = (input + output tokens) / duration.
• The duration is the wall time of the whole run, from the start of the benchmark to the last result (vllm/benchmarks/serve.py:1048, vllm/benchmarks/serve.py:1123).
• Mean, median, standard deviation and the selected percentiles, with np.percentile (vllm/benchmarks/serve.py:782-806).

Goodput. A request is good when every given SLO value is greater than or equal to its measured value (vllm/benchmarks/serve.py:669-692). Request goodput = good requests / duration. A request with 1 output token has TPOT 0 for this check.

    if goodput_config_dict:
        valid_metrics = []
        slo_values = []

        if "ttft" in goodput_config_dict:
            valid_metrics.append(ttfts)
            slo_values.append(
                goodput_config_dict["ttft"] / MILLISECONDS_TO_SECONDS_CONVERSION
            )
        if "tpot" in goodput_config_dict:
            valid_metrics.append(all_tpots)
            slo_values.append(
                goodput_config_dict["tpot"] / MILLISECONDS_TO_SECONDS_CONVERSION
            )
        if "e2el" in goodput_config_dict:
            valid_metrics.append(e2els)
            slo_values.append(
                goodput_config_dict["e2el"] / MILLISECONDS_TO_SECONDS_CONVERSION
            )

        for req_metric in zip(*valid_metrics):
            is_good_req = all([s >= r for s, r in zip(slo_values, req_metric)])
            if is_good_req:
                good_completed += 1

The main flags.

An example command (not in the repo):

vllm bench serve --model meta-llama/Llama-3.1-8B-Instruct \
  --request-rate 4 --num-prompts 200 \
  --metric-percentiles 50,99 --goodput ttft:500 tpot:50

This is a made-up workload (panel 3 of the diagram). The run lasts 8.0 s. The SLO is --goodput ttft:500 tpot:50.

Check one row: E2E of A = 200 + (101 − 1) × 20 = 2,200 ms.

• Request throughput = 5 / 8.0 = 0.625 req/s.
• Output token throughput = 455 / 8.0 = 56.9 tok/s. Total token throughput = (4,600 + 455) / 8.0 = 631.9 tok/s.
• Request goodput = 3 / 8.0 = 0.375 req/s. Throughput says 0.625, but only 60 % of the requests met the SLO.
• TTFT: sorted 120, 150, 200, 300, 650. Mean 284 ms, median 200 ms. np.percentile interpolates in a straight line by default. The p99 is at index 0.99 × 4 = 3.96, so p99 = 300 + 0.96 × (650 − 300) = 636 ms.
• TPOT: E has 1 output token, so only A to D count: 20, 22, 25, 55. Mean 30.5 ms, median 23.5 ms. The p99 is at index 0.99 × 3 = 2.97, so p99 = 25 + 0.97 × (55 − 25) = 54.1 ms.
• Average concurrency = the sum of all E2E times / duration = 14.57 s / 8.0 s = 1.8 requests in flight.

The mean TTFT (284 ms) is below the SLO, but one request in five missed it. This is why SLOs use percentiles or goodput, not means.

The vLLM scheduler has no prefill phase and no decode phase. Each step gives tokens to requests until their computed tokens catch up with their known tokens (vllm/v1/core/sched/scheduler.py:624-633). So one step can hold decode tokens and prefill chunks together.

1. The step starts with the token budget max_num_scheduled_tokens (vllm/v1/core/sched/scheduler.py:642). This is normally max_num_batched_tokens (vllm/v1/core/sched/scheduler.py:138-142).
2. It first serves the running requests (vllm/v1/core/sched/scheduler.py:713-715).
3. It then takes waiting requests with the rest of the budget (vllm/v1/core/sched/scheduler.py:956). Chunked prefill is on by default for generative models that are not encoder-decoder models (vllm/engine/arg_utils.py:3031-3035). So a long prompt takes only the rest of the budget.

Defaults for vllm serve (usage context OPENAI_API_SERVER) on a GPU depend on the GPU memory (vllm/engine/arg_utils.py:2960-2993). They apply only when you do not set the flag. --performance-mode throughput doubles both values (vllm/engine/arg_utils.py:3171-3175). TPU and CPU platforms have other values (vllm/engine/arg_utils.py:2995-3024).

The vLLM optimization guide gives the direction. A smaller max_num_batched_tokens (for example 2,048) gives a better ITL, and a larger one gives a better TTFT (docs/configuration/optimization.md:64-65). max_num_seqs limits the requests that run at the same time, so it limits the batch and the concurrency inside the engine (vllm/v1/core/sched/scheduler.py:130-137). The page Tuning vLLM: throughput and latency knobs, their real defaults and what each one does explains both with the code.

With --enable-per-request-metrics (default off, vllm/entrypoints/common/cli_args.py:150-151), the response has a metrics object. In streaming, it comes in the final usage chunk. The server sends that chunk only when usage is on: stream_options.include_usage or --enable-force-include-usage. The server leaves the object out when n is more than 1 (vllm/entrypoints/openai/chat_completion/serving.py:873-885). The flag needs engine statistics: with --disable-log-stats, start-up fails (vllm/entrypoints/common/cli_args.py:536-542).

The other fields: queue_time_ms = scheduled_ts − queued_ts; generation_time_ms = decode interval; mean_itl_ms = decode interval / (tokens − 1); tokens_per_second = all output tokens / inference interval (vllm/entrypoints/generate/base/serving.py:76-89). vllm bench serve reads queue_time_ms and time_to_first_token_ms from this object when the server sends it (vllm/benchmarks/lib/endpoint_request_func.py:110-125).