Prefix caching: the idea

The numbers are made up, but the arithmetic follows the code. The block size is 16, the default (vllm/config/cache.py:182, vllm/config/cache.py:459-460). Some attention backends need another size, and then the platform changes it (vllm/platforms/interface.py:771-781). With one KV cache group, vLLM hashes at the same size as the block size (vllm/v1/core/kv_cache_utils.py:840-858).

Turn 1 (request A). The system prompt, the first user message and the chat template give 75 prompt tokens. The cache is empty, so the hit is 0 and the engine computes all 75 tokens.

• 75 tokens = 4 full blocks (64 tokens) + 11 tokens in block 4. So request A has 4 block hashes after its prompt (vllm/v1/core/kv_cache_utils.py:990-993).
• The model answers with 45 tokens. The request then has 120 tokens. The block hasher runs again when new full blocks appear, so blocks 4, 5 and 6 also get hashes (vllm/v1/core/kv_cache_utils.py:966-972). Request A ends with 7 hashed blocks (112 tokens) and one block with 8 tokens and no hash.
• When request A finishes, its blocks go back to the free queue with their hashes.

Turn 2 (request B). The client sends the system prompt, user message 1, the answer 1 and user message 2. The chat template renders this as 150 tokens. Assume that tokens 0 to 117 are the same as in request A. The template adds tokens after the answer, so token 118 is the first new token.

So request B gets 7 × 16 = 112 cached tokens and computes 150 − 112 = 38 tokens. That is 75 % less prefill work for this turn. With a long system prompt, the saving is often larger.

The metric for both turns. vLLM counts the prompt tokens of each lookup as queries and the matched tokens as hits (next deep dive). Queries = 75 + 150 = 225 tokens. Hits = 0 + 112 = 112 tokens. The hit rate is 112 / 225 = 49.8 %.

One edge case. Assume a prompt of exactly 64 tokens that is fully in the cache. The model must still compute the last prompt token, because it needs the logits of that token. So the maximum hit is num_tokens − 1 = 63 tokens (vllm/v1/core/kv_cache_manager.py:289-300). The lookup checks only 63 // 16 = 3 blocks (vllm/v1/core/single_type_kv_cache_manager.py:793). So the hit is 48 tokens, and the engine computes the last 16 tokens again.

The engine core makes one hasher function at start-up. The function gets the hash block size and the hash function (vllm/v1/engine/core.py:216-225). The engine core then computes the block hashes of each new request.

    def request_block_hasher(request: Request) -> list[BlockHash]:
        start_token_idx = len(request.block_hashes) * hash_block_size
        num_tokens = request.num_tokens

        if start_token_idx + hash_block_size > num_tokens:
            # Early stop when there no new full blocks created.
            return []

        curr_mm_idx = 0
        mm_features = request.mm_features
        if start_token_idx > 0 and mm_features:
            last_mm_pos = mm_features[-1].mm_position
            if last_mm_pos.offset + last_mm_pos.length > start_token_idx:
                curr_mm_idx, _ = get_mm_features_in_window(
                    mm_features,
                    start_token_idx,
                    start_token_idx + hash_block_size,
                )

        prev_block_hash_value = (
            request.block_hashes[-1] if request.block_hashes else None
        )
        new_block_hashes: list[BlockHash] = []
        while True:
            end_token_idx = start_token_idx + hash_block_size
            if end_token_idx > num_tokens:
                # We only hash full blocks
                break

            # MM and LoRA requests need extra keys for block-hash computation.
            extra_keys, curr_mm_idx = generate_block_hash_extra_keys(
                request, start_token_idx, end_token_idx, curr_mm_idx
            )

            # Compute the hash of the current block
            block_tokens = request.all_token_ids[start_token_idx:end_token_idx]
            block_hash = hash_block_tokens(
                caching_hash_fn, prev_block_hash_value, block_tokens, extra_keys
            )

            new_block_hashes.append(block_hash)
            start_token_idx += hash_block_size
            prev_block_hash_value = block_hash

        return new_block_hashes

1. Lines 967-972. The hasher starts after the blocks that already have a hash. If no new full block exists, it returns an empty list.
2. Lines 974-983. For multimodal requests, it finds the first multimodal item that can touch the next block.
3. Lines 985-987. The parent of the first new block is the last hash of the request, or None for block 0.
4. Lines 988-993. The loop stops before a block that is not full: "We only hash full blocks".
5. Lines 996-998. It makes the extra keys of this block (LoRA, multimodal items, cache salt, prompt embeddings).
6. Lines 1001-1004. It hashes the parent hash, the 16 token ids and the extra keys.
7. Lines 1006-1008. The new hash becomes the parent of the next block. This is the chain.

The value of block −1. Block 0 has no parent, so hash_block_tokens() uses NONE_HASH (vllm/v1/core/kv_cache_utils.py:731-732). init_none_hash() derives it from a seed (vllm/v1/core/kv_cache_utils.py:146-159). PYTHONHASHSEED wins if it is set. Otherwise sha256 uses the fixed seed "vllm-none-hash", and the xxhash options use random bytes (vllm/v1/core/kv_cache_utils.py:117-130). With a random seed, two vLLM instances compute different hashes for the same tokens.

The hash functions. Four names are valid (vllm/config/cache.py:174), and get_hash_fn_by_name() maps them to functions (vllm/utils/hashing.py:84-103):

generate_block_hash_extra_keys() collects the extra keys of one block (vllm/v1/core/kv_cache_utils.py:637-675):

• LoRA. ("lora", lora_name, lora_path) for every block of a LoRA request (vllm/v1/core/kv_cache_utils.py:590-607). The path is in the key, so a name that points to a new adapter path does not reuse old blocks.
• Multimodal items. ("mm", identifier, offset) for each image, audio or video item that overlaps the block. The offset is the start of the item minus the start of the block. It is negative when the item starts in an earlier block (vllm/v1/core/kv_cache_utils.py:564-574).
• Cache salt. ("cache_salt", salt), but only for block 0 (vllm/v1/core/kv_cache_utils.py:659-663). Block 0 is enough: every later hash depends on the hash of block 0.
• Prompt embeddings. A SHA-256 of the embedding tensor of the block, when the request sends embeddings instead of token ids (vllm/v1/core/kv_cache_utils.py:610-634).

The cache_salt request field. The chat completions request has an optional cache_salt string (vllm/entrypoints/openai/chat_completion/protocol.py:499-511). Its description says that the salt should be random and long, for example 256 bits in base64. A validator checks it before the request runs (vllm/entrypoints/openai/chat_completion/protocol.py:560-566). The salt must be a non-empty string of at most 128 characters, without @, /, \ or NUL. Otherwise the server returns HTTP 400 (vllm/entrypoints/generate/base/protocol.py:39-58).

{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [{"role": "user", "content": "Hi"}],
  "cache_salt": "Zk9hc2l2ZS1yYW5kb20tc2FsdC1mb3ItdGVuYW50LTQy"
}

Why a salt. A hit makes TTFT shorter. An attacker could send a guessed prompt and measure TTFT to learn whether another user sent that prompt. The design document gives this reason (docs/design/prefix_caching.md:87). Requests with different salts never share blocks.

The block pool keeps all free blocks in one doubly linked list, FreeKVCacheBlockQueue. The class comment gives the order: the least recently used block is at the front. For two blocks that a request freed at the same time, the block with more hashed tokens (the tail of the chain) comes first (vllm/v1/core/kv_cache_utils.py:258-279).

1. Free. A request frees its blocks in reverse order, so its tail block is first (vllm/v1/core/single_type_kv_cache_manager.py:597-598). free_blocks() puts blocks without a hash at the front of the queue, and blocks with a hash at the end (vllm/v1/core/block_pool.py:776-807). So the pool reuses blocks without a hash first, and it keeps cached blocks as long as it can.
2. Hit. When a request hits a free cached block, touch() removes the block from the free queue and increments its reference count (vllm/v1/core/block_pool.py:752-770). Several running requests can share one block.
3. Allocate. get_new_blocks() pops blocks from the front of the queue (vllm/v1/core/block_pool.py:678-681). If such a block still has a hash, _maybe_evict_cached_block() removes the hash from the cache (vllm/v1/core/block_pool.py:729-750). This is the eviction.

There is no time limit and no separate cache size. The cache is simply the set of free blocks that still have a hash. The more free KV memory the GPU has, the longer cached blocks survive.

The scheduler records one lookup per admitted request (vllm/v1/core/sched/scheduler.py:1378-1382). The KV cache manager adds the prompt tokens as queries and the matched tokens as hits (vllm/v1/core/kv_cache_manager.py:253-262). It skips requests that did not look up the cache. The counters vllm:prefix_cache_queries and vllm:prefix_cache_hits also skip a request that was preempted before. Its new lookup goes to separate preempted_* fields, which no logger exports (vllm/v1/metrics/stats.py:146-151).

A hit rate for a dashboard is rate(vllm:prefix_cache_hits[5m]) / rate(vllm:prefix_cache_queries[5m]). The rate is a fraction of tokens, not of requests.

Requests that skip the cache. A request that asks for prompt logprobs does not read the cache by default, because the hit tokens would have no logprobs (vllm/sampling_params.py:625-632). The lookup also does not run when prefix caching is off (vllm/v1/core/kv_cache_manager.py:249-251).

Reset. The route POST /reset_prefix_cache exists, but only as a development route (vllm/entrypoints/serve/dev/cache/api_router.py:20). The API server adds the dev routes only when VLLM_SERVER_DEV_MODE is set (vllm/entrypoints/launchers/api_server/routers.py:46-49).