Benchmarking vLLM: vllm bench serve, throughput, latency and startup

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

To tune a server, you need numbers that you can repeat. vllm bench has several subcommands. Only one of them talks to a running server:

• vllm bench serve sends HTTP requests to a server that you started with vllm serve. It measures what a client sees. The metrics are time to first token (TTFT), time per output token (TPOT), inter-token latency (ITL), end-to-end latency (E2EL), throughput and goodput (Serving metrics: TTFT, TPOT, ITL, throughput, goodput).
• vllm bench throughput, latency and startup start their own engine in the same command, with the offline LLM class. They measure the engine without HTTP.

The client controls the load: how many requests, how long each prompt is, how fast they arrive, and how many run at once. The server's own metrics measure the same requests from the inside (Metrics and observability: how to watch a vLLM server). The two views differ by design, and this page shows where.

An everyday comparison. Think of a test shopper. The shopper starts a stopwatch at the shop door and stops it when the bag is in the hand. The shop's own records start when the order reaches the counter. Both are correct; they time different things. The comparison stops at the crowd. The test shopper of vLLM can be a thousand shoppers at once, with a random rhythm of arrivals that you choose.

The picture#

Read panel 1 from left to right. The client clock starts when the POST leaves the client, before the server parses HTTP. The server clocks start only when rendering starts. Panel 2 is the summary to keep: compare a client number only with a client number.

In the code#

Step 1: the command and the request function#

vllm bench serve is a thin subcommand: it adds the arguments of vllm/benchmarks/serve.py and calls its main (vllm/entrypoints/cli/benchmark/serve.py:10-21). The default backend is openai and the default endpoint is /v1/completions (vllm/benchmarks/serve.py:1647-1668). The backend picks a request function; openai-chat sends chat requests, and there are functions for embeddings and rerank servers (vllm/benchmarks/lib/endpoint_request_func.py:1155-1172). Without --model, the client asks the server's /v1/models for the first model (vllm/benchmarks/serve.py:2126-2131, vllm/benchmarks/serve.py:170-176).

Each request is a stream with usage in the last chunk, so the client gets the server's token count (vllm/benchmarks/lib/endpoint_request_func.py:226-238). The client takes its time points here:

                            if choices := data.get("choices"):
                                # Note that text could be empty here
                                # e.g. for special tokens
                                text = choices[0].get("text")
                                timestamp = time.perf_counter()
                                # First token
                                if not first_chunk_received:
                                    first_chunk_received = True
                                    output.ttft = timestamp - st

                                # Decoding phase
                                else:
                                    output.itl.append(timestamp - most_recent_timestamp)

                                most_recent_timestamp = timestamp
                                generated_text += text or ""
                            elif usage := data.get("usage"):
                                output.output_tokens = usage.get("completion_tokens")
                                if (pt := usage.get("prompt_tokens")) is not None:
                                    output.prompt_len = pt

1. The clock starts with time.perf_counter() just before the POST (vllm/benchmarks/lib/endpoint_request_func.py:248-252).
2. Line 286: TTFT is the time of the first SSE chunk that has choices, even when its text is empty.
3. Line 290: each later chunk with choices adds one ITL value. One chunk can carry several tokens (for example with --stream-interval), so the client ITL is per chunk, not per token.
4. Line 295: the usage chunk gives completion_tokens; it does not move the end time.
5. E2EL (latency) is the time of the last chunk with choices (vllm/benchmarks/lib/endpoint_request_func.py:307). A request with no such chunk is a failure (vllm/benchmarks/lib/endpoint_request_func.py:298-305).

Step 2: datasets#

--dataset-name picks where the prompts come from; the default is random (vllm/benchmarks/datasets/datasets.py:1593-1612):

The random dataset makes token ids from a seeded generator, decodes them and encodes them again, so the prompt has the asked length (vllm/benchmarks/datasets/datasets.py:556-579). --seed (default 0) seeds it and the arrival times (vllm/benchmarks/datasets/datasets.py:1585, vllm/benchmarks/serve.py:2060-2061). --num-prompts defaults to 1,000 (vllm/benchmarks/datasets/datasets.py:70).

Step 3: request rate, burstiness and max concurrency#

--request-rate is in requests per second; the default inf sends all requests at time 0 (vllm/benchmarks/serve.py:1745-1753). For a finite rate, the client draws the gap before each request from a gamma distribution:

            if current_request_rate == float("inf"):
                delay_ts.append(0)
            elif burstiness == float("inf"):
                # when burstiness tends to infinity, the delay time becomes constant
                # and tends to the inverse of the request rate
                delay_ts.append(1.0 / current_request_rate)
            else:
                theta = 1.0 / (current_request_rate * burstiness)

                # Sample the request interval from the gamma distribution.
                # If burstiness is 1, it follows exponential distribution.
                delay_ts.append(np.random.gamma(shape=burstiness, scale=theta))

        # Calculate the cumulative delay time from the first sent out requests.
        for i in range(1, len(delay_ts)):
            delay_ts[i] += delay_ts[i - 1]
        if ramp_up_strategy is None and delay_ts[-1] != 0:
            # When ramp_up_strategy is not set, we assume the request rate is fixed
            # and all requests should be sent in target_total_delay_s, the following
            # logic would re-scale delay time to ensure the final delay_ts
            # align with target_total_delay_s.
            #
            # NOTE: If we simply accumulate the random delta values
            # from the gamma distribution, their sum would have 1-2% gap
            # from target_total_delay_s. The purpose of the following logic is to
            # close the gap for stabilizing the throughput data
            # from different random seeds.
            target_total_delay_s = total_requests / request_rate
            normalize_factor = target_total_delay_s / delay_ts[-1]
            delay_ts = [delay * normalize_factor for delay in delay_ts]

• --burstiness is the shape of the gamma distribution, default 1.0 (vllm/benchmarks/serve.py:1754-1765). The scale is 1 / (rate × burstiness), so the mean gap is always 1 / rate. Burstiness 1 gives exponential gaps (a Poisson process). Below 1, gaps vary more (bursts); above 1, they are more even.
• Lines 513-526: without a ramp-up, the client then scales all gaps so that the last request starts exactly at num_prompts / rate. The comment says that this keeps the throughput stable across seeds.
• --max-concurrency N wraps each request in an asyncio.Semaphore(N) (vllm/benchmarks/serve.py:1001-1013). Requests that wait for the semaphore start their TTFT clock only when they get it; the wait is kept as client_queue_time (vllm/benchmarks/serve.py:1014-1016). With a rate and a concurrency limit, the real rate can be lower than the asked rate (vllm/benchmarks/serve.py:1679-1690).
• --ramp-up-strategy linear|exponential with start and end rates changes the rate during the run (vllm/benchmarks/serve.py:1965-1987).

Step 4: warm-up and the ready check#

Two start steps are off by default. --ready-check-timeout-sec defaults to 0, so the client skips the initial test request and prints "Skipping endpoint ready check." (vllm/benchmarks/serve.py:1989-1994, vllm/benchmarks/serve.py:908-925). --num-warmups defaults to 0 (vllm/benchmarks/serve.py:1780-1785). A first request after a server start can pay one-time costs, so a run without warm-up includes them in its first results.

Step 5: how the metrics are computed#

After all requests finish, calculate_metrics builds the result. TPOT is computed per request:

            actual_output_lens.append(output_len)
            total_input += outputs[i].prompt_len
            tpot = 0.0
            if output_len > 1:
                latency_minus_ttft = outputs[i].latency - outputs[i].ttft
                tpot = latency_minus_ttft / (output_len - 1)
                tpots.append(tpot)
            # Note: if output_len <= 1, we regard tpot as 0 for goodput
            all_tpots.append(tpot)
            itls += outputs[i].itl
            ttfts.append(outputs[i].ttft)

• TPOT = (E2EL − TTFT) / (output tokens − 1), and only requests with more than one output token enter the TPOT statistics (lines 655-658). For goodput, a one-token request has TPOT 0 (line 660).
• The output token count comes from the server's usage; without it, the client tokenizes the text (vllm/benchmarks/serve.py:635-650).
• Throughputs divide by the benchmark duration, from the start of the send loop to the end of the last request (vllm/benchmarks/serve.py:773-781, vllm/benchmarks/serve.py:1048, vllm/benchmarks/serve.py:1123).
• Mean, median, standard deviation and percentiles use NumPy over all requests (TTFT, TPOT, E2EL) or over all chunk gaps (ITL) (vllm/benchmarks/serve.py:782-806). --metric-percentiles defaults to 99 (vllm/benchmarks/serve.py:1858-1866); --percentile-metrics defaults to ttft,tpot,itl for generative models (vllm/benchmarks/serve.py:2255).
• --goodput ttft:500 tpot:50 sets SLOs in milliseconds (vllm/benchmarks/serve.py:1867-1878). A request is good when each given SLO is at least its measured value. Request goodput = good requests / duration (vllm/benchmarks/serve.py:669-692, vllm/benchmarks/serve.py:779).

Step 6: client numbers vs server numbers#

Run vllm bench serve against a server and watch its /metrics at the same time: the numbers differ. These are the reasons, each from the code:

With --enable-per-request-metrics on the server, the client also stores the server's queue_time_ms and time_to_first_token_ms per request in its result data (vllm/benchmarks/lib/endpoint_request_func.py:110-125, vllm/benchmarks/serve.py:391-403). That server TTFT is the prefill interval only (Metrics and observability: how to watch a vLLM server). With speculative decoding, the client reads the server's vllm:spec_decode_* counters before and after the run and reports the acceptance (vllm/benchmarks/serve.py:996, vllm/benchmarks/serve.py:210-260).

Step 7: vllm bench throughput#

vllm bench throughput measures offline throughput: it builds an LLM from the engine flags and sends all prompts in one generate() call (vllm/benchmarks/throughput.py:37-48, vllm/benchmarks/throughput.py:129-151). It prints requests/s, total tokens/s and output tokens/s (vllm/benchmarks/throughput.py:1108-1114). Without --dataset-path, it uses the random dataset and needs --input-len or --random-input-len (vllm/benchmarks/throughput.py:642-650). --async-engine uses an in-process AsyncLLM instead (vllm/benchmarks/throughput.py:863-867, vllm/benchmarks/throughput.py:308-314). There is no HTTP, no SLO and no arrival process: all requests wait at time 0.

Step 8: vllm bench latency#

vllm bench latency measures the time of one batch. The batch has --batch-size (default 8) random prompts of --input-len (32) tokens. Each prompt gets exactly --output-len (128) tokens, because the command sets ignore_eos (vllm/benchmarks/latency.py:35-37, vllm/benchmarks/latency.py:96-103). It runs 10 warm-up iterations and 30 measured ones, and prints the mean and the 10th to 99th percentiles of the batch time (vllm/benchmarks/latency.py:45-53, vllm/benchmarks/latency.py:158-168).

Step 9: vllm bench startup#

vllm bench startup measures how long LLM(...) takes to start, cold and warm (vllm/benchmarks/startup.py:3-9). A cold run uses a new temporary VLLM_CACHE_ROOT and a fresh Inductor cache; a warm run reuses the caches (vllm/benchmarks/startup.py:109-128). Each run is a new subprocess (vllm/benchmarks/startup.py:133-147). The defaults are 3 cold runs, 1 warm-up and 3 warm runs (vllm/benchmarks/startup.py:193-210). It reports the start-up time and the compilation time of each run.

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

vllm bench serve --model meta-llama/Llama-3.1-8B-Instruct --dataset-name random \
  --random-input-len 1024 --random-output-len 128 --ignore-eos --num-prompts 1000 \
  --request-rate 20 --num-warmups 20 --metric-percentiles 50,99 --goodput ttft:500 tpot:50

============ Serving Benchmark Result ============
Successful requests:                     1000
Failed requests:                         0
Request rate configured (RPS):           20.00
Benchmark duration (s):                  52.10
Total input tokens:                      1024000
Total generated tokens:                  128000
Request throughput (req/s):              19.19
Request goodput (req/s):                 17.60
Output token throughput (tok/s):         2456.81
Peak output token throughput (tok/s):    3120.00
Peak concurrent requests:                61.00
Total token throughput (tok/s):          22111.32
---------------Time to First Token----------------
Mean TTFT (ms):                          182.40
Median TTFT (ms):                        140.10
P50 TTFT (ms):                           140.10
P99 TTFT (ms):                           690.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.30
Median TPOT (ms):                        23.90
P50 TPOT (ms):                           23.90
P99 TPOT (ms):                           38.20
---------------Inter-token Latency----------------
Mean ITL (ms):                           24.20
Median ITL (ms):                         19.80
P50 ITL (ms):                            19.80
P99 ITL (ms):                            95.70
==================================================

What can go wrong#

• Huge TTFT at --request-rate inf. All requests arrive at time 0, so most of them wait in the queue (Step 3). Use a finite rate or --max-concurrency.
• Output tokens below the asked length. The model stopped at an end token; use --ignore-eos (Step 2).
• The first run is slower than the next ones. No warm-up by default (Step 4).
• Client ITL is much larger than Prometheus ITL. The server sends several tokens per chunk (--stream-interval), or the client process is busy (Step 1, Step 6).
• All requests fail with "Never received a valid chunk". The endpoint does not stream choices, for example the wrong --backend for --endpoint (vllm/benchmarks/lib/endpoint_request_func.py:298-305).
• vllm bench throughput is faster than the server. It has no HTTP, no arrival process, and a larger default token budget (note in Step 9).

Why it matters. A benchmark number is only useful when you know its start point, its clock and its load. With that, you can compare two configurations fairly and explain the gap between the client view and the server view.