Prefix caching in the engine core: hashes, lookup, caching, eviction and KV events

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

The page Prefix caching: the idea teaches the idea: why chat traffic repeats, the hash chain, the extra keys, two chat turns and the metrics. Read it first. This page shows who does each part in the engine core, when, and with which data. It also covers what the basics page leaves out. That is the hash types, the lookup code and the moment a block gets its hash. It is also the reset route, external hits from a KV connector, and KV events.

Four parts work together, all in the EngineCore process:

1. The hasher. When a request arrives, the input thread computes one block hash per full block of its tokens. When the request gets new tokens, the hasher adds the hashes of new full blocks.
2. The lookup. When the scheduler admits a request, the KV cache manager walks the hashes from block 0 and stops at the first miss. It never looks past num_tokens − 1 tokens (cache hit cap).
3. The cache. When allocate_slots() gives a request its blocks, the blocks that will be full after this step get their hash. From then on, other requests can hit them.
4. The eviction. A freed block keeps its hash in the free block queue. Normally its hash goes away only when the pool hands the block out again, or when somebody resets the cache. A failed load from a KV connector can also evict hashes (vllm/v1/core/sched/scheduler.py:3515-3516).

An everyday comparison. Think of a library that keeps photocopies of book chapters. Each copy has a label that names the book and all earlier chapters. A reader asks for chapters 1 to 5. The desk checks the labels in order and stops at the first chapter that has no copy. Old copies stay on a "free" shelf, and the library recycles the oldest copy only when it needs paper. The comparison stops at the label: vLLM does not compare text. Two blocks match when their hashes are equal, and the hash includes every earlier token.

The picture#

Read the panels from top to bottom. Panel 2 is get_computed_blocks(): a miss ends the walk, because every later hash includes the missed one. Panel 3 is the first part of allocate_slots(). touch() runs before get_new_blocks(), so the request cannot pop and evict its own hit blocks.

The explorer follows these rules of the real code:

1. Only full blocks get a hash. The hash of block i uses the hash of block i − 1, and block 0 uses NONE_HASH (vllm/v1/core/kv_cache_utils.py:988-1008, vllm/v1/core/kv_cache_utils.py:731-737).
2. The lookup checks at most (num_tokens − 1) // block_size hashes and stops at the first miss (vllm/v1/core/kv_cache_manager.py:295-300, vllm/v1/core/single_type_kv_cache_manager.py:791-799).
3. A request starts only if its new blocks plus its free hit blocks fit in the free queue (vllm/v1/core/single_type_kv_cache_manager.py:256-266, vllm/v1/core/kv_cache_manager.py:575-579).
4. touch() adds 1 to ref_cnt and takes a free hit block out of the queue (vllm/v1/core/block_pool.py:761-768). Only then does get_new_blocks() pop new blocks from the front and evict their old hashes (vllm/v1/core/block_pool.py:681-691).
5. The blocks that are full now get their hash; an equal hash on a second block is kept as a duplicate (vllm/v1/core/block_pool.py:273-299, vllm/v1/core/block_pool.py:48-52).
6. A finished request frees its blocks in reverse order. At ref_cnt 0, a block without a hash goes to the front, a block with a hash to the back (vllm/v1/core/single_type_kv_cache_manager.py:597-598, vllm/v1/core/block_pool.py:795-807).
7. Block 0 is the null block; the pool never hands it out (vllm/v1/core/block_pool.py:184-188).

In the code#

Step 1: the hasher is made once, and runs when a request arrives#

EngineCore.__init__ first resolves the hash granularity, hash_block_size. With one KV cache group, it is the block size (vllm/v1/engine/core.py:173-175, vllm/v1/core/kv_cache_utils.py:840-858). Then it makes the request hasher, but only when prefix caching or a KV connector is on:

        self.request_block_hasher: Callable[[Request], list[BlockHash]] | None = None
        if vllm_config.cache_config.enable_prefix_caching or kv_connector is not None:
            caching_hash_fn = get_hash_fn_by_name(
                vllm_config.cache_config.prefix_caching_hash_algo
            )
            init_none_hash(caching_hash_fn)

            self.request_block_hasher = get_request_block_hasher(
                hash_block_size, caching_hash_fn
            )

1. Line 217. No prefix caching and no connector: request_block_hasher stays None, and requests get no hashes at all.
2. Lines 218-221. The hash function comes from --prefix-caching-hash-algo. init_none_hash() sets the module-level NONE_HASH for this process.
3. Lines 223-225. get_request_block_hasher() returns a closure over the block size and the hash function (vllm/v1/core/kv_cache_utils.py:953-1012).

Each new Request gets this function and calls it in its constructor (vllm/v1/engine/core.py:1128, vllm/v1/request.py:225-230). In vllm serve, the input thread of the engine core process builds the Request while the main thread runs the model (vllm/v1/engine/core.py:1114-1119, vllm/v1/engine/core.py:1990-1993). So the hashing of a long prompt does not delay the current step.

When the scheduler appends a sampled token, the request runs the hasher again (vllm/v1/core/sched/scheduler.py:2598-2599, vllm/v1/request.py:272-288). The hasher starts after the last hash and returns an empty list until a new block is full (vllm/v1/core/kv_cache_utils.py:967-972). So request.block_hashes always covers every full block of the known tokens.

Step 2: the hash types#

# BlockHash represents the hash of a single KV-cache block used for
# prefix caching.  Treating it as a distinct type from `bytes` helps
# catch accidental misuse when passing around raw byte strings.
BlockHash = NewType("BlockHash", bytes)

# `BlockHashWithGroupId` combines a `BlockHash` with its KV cache group ID.
# It is represented as raw bytes for compactness and efficiency. The helper
# functions below pack/unpack the `BlockHash` and group id into/from the key.
BlockHashWithGroupId = NewType("BlockHashWithGroupId", bytes)

# ExternalBlockHash is used for reproducible prefix-cache block hashing.
# It's a union of `bytes` and `int` to keep backward compatibility
# after we default block hashing to use sha256 bytes.
ExternalBlockHash: TypeAlias = bytes | int


def make_block_hash_with_group_id(
    block_hash: BlockHash, group_id: int
) -> BlockHashWithGroupId:
    """Pack a `BlockHash` and group id into a `BlockHashWithGroupId`.

    The group id is encoded using 4 bytes in big-endian order and appended to
    the block hash bytes.  This representation avoids creating tuples while
    still allowing us to recover both components when needed.
    """
    return BlockHashWithGroupId(block_hash + group_id.to_bytes(4, "big", signed=False))


def get_block_hash(key: BlockHashWithGroupId) -> BlockHash:
    """Extract the `BlockHash` from a `BlockHashWithGroupId`."""
    return BlockHash(key[:-4])


def get_group_id(key: BlockHashWithGroupId) -> int:
    """Extract the group id from a `BlockHashWithGroupId`."""
    return int.from_bytes(key[-4:], "big", signed=False)


def maybe_convert_block_hash(hash_bytes: BlockHash) -> ExternalBlockHash:
    if not envs.VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES:
        return hash_bytes
    return int.from_bytes(hash_bytes, byteorder="big") & ((1 << 64) - 1)

• BlockHash is the raw digest of one block: 32 bytes for sha256, 16 bytes for the xxHash 128-bit options (vllm/utils/hashing.py:41, vllm/utils/hashing.py:69).
• BlockHashWithGroupId appends the KV cache group id as 4 big-endian bytes. The hash map uses this key, so the same tokens in two groups are two entries (line 69).
• ExternalBlockHash is what KV events publish. By default VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES=1 turns the digest into a 64-bit integer: the low 64 bits of the big-endian number (line 85, vllm/envs.py:2022-2024).

Step 3: the hash function and NONE_HASH#

The four names of --prefix-caching-hash-algo and their serialization are in the basics page (Prefix caching: the idea, deep dive "The hash chain in the code"). In short: sha256 (the default) pickles the tuple (parent, tokens, extra_keys); the _cbor options use canonical CBOR, so other languages can compute the same hash (vllm/utils/hashing.py:26-81).

The comment above NONE_HASH gives the reason for the two seeds. A cryptographic hash can use a fixed, public seed, so independent processes compute equal hashes and can share a cache. A non-cryptographic hash keeps a secret random seed, because a known seed would let an attacker build colliding blocks (vllm/v1/core/kv_cache_utils.py:90-101). PYTHONHASHSEED overrides both (vllm/v1/core/kv_cache_utils.py:125-130).

Step 4: extra keys#

generate_block_hash_extra_keys() joins four lists in a fixed order: LoRA, multimodal items, cache salt, prompt-embedding digests. Without any key it returns None, so a plain text block hashes (parent, tokens, None):

    mm_extra_keys: list[Any]
    mm_extra_keys, new_start_mm_idx = _gen_mm_extra_hash_keys(
        request, start_token_idx, end_token_idx, start_mm_idx
    )
    lora_extra_keys = _gen_lora_extra_hash_keys(request)
    cache_salt_keys: list[tuple[str, str]] = (
        [("cache_salt", request.cache_salt)]
        if (start_token_idx == 0 and request.cache_salt)
        else []
    )
    prompt_embeds_keys = _gen_prompt_embeds_extra_hash_keys(
        request, start_token_idx, end_token_idx
    )

    extra_keys: list[Any] = (
        lora_extra_keys + mm_extra_keys + cache_salt_keys + prompt_embeds_keys
    )

    if not extra_keys:
        return None, new_start_mm_idx

    return tuple(extra_keys), new_start_mm_idx

The basics page explains each key (Prefix caching: the idea, deep dive "Extra keys"). KV events publish the keys without their tags: a bare LoRA name, a bare salt, (identifier, offset) pairs for images (vllm/v1/core/kv_cache_utils.py:678-704).

Step 5: the lookup#

The scheduler asks for the hit only in the waiting pass, and only for a request with num_computed_tokens == 0: a new or a preempted request (vllm/v1/core/sched/scheduler.py:1016-1024). KVCacheManager.get_computed_blocks() does the work:

        # We skip finding the prefix cache hit when prefix caching is
        # disabled or the request is marked as skipping kv cache read
        # (which happens when the request requires prompt logprobs
        # or calls a pooling model with all pooling).
        if not self.prefix_cache_lookup_enabled(request):
            return self.empty_kv_cache_blocks, 0, 0

        # NOTE: When all tokens hit the cache, we must recompute the last token
        # to obtain logits. Thus, set max_cache_hit_length to prompt_length - 1.
        # This can trigger recomputation of an entire block, rather than just
        # the single last token, because allocate_slots() requires
        # num_computed_tokens to be block-size aligned. Removing this limitation
        # could slightly improve performance in the future.
        max_cache_hit_length = request.num_tokens - 1
        computed_blocks, num_new_computed_tokens, num_uncached = (
            self.coordinator.find_longest_cache_hit(
                request.block_hashes, max_cache_hit_length
            )
        )

1. Lines 286-287. No lookup when prefix caching is off, or when the request sets skip_reading_prefix_cache (vllm/v1/core/kv_cache_manager.py:249-251). Sampling parameters set it for prompt logprobs; pooling parameters can set it too (vllm/sampling_params.py:625-632, vllm/v1/request.py:314-325).
2. Line 295. The cache hit cap: the last token must be computed, so it gives logits.
3. Lines 296-300. The coordinator finds the hit. With one group, UnitaryKVCacheCoordinator calls the manager of that group directly (vllm/v1/core/kv_cache_coordinator.py:574-591).

For full attention, the walk is short:

        computed_blocks: tuple[list[KVCacheBlock], ...] = tuple(
            [] for _ in range(len(kv_cache_group_ids))
        )
        # Phase 1: longest run of cached full blocks from the start. A missing
        # block implies every later block misses too (chained hashes).
        for block_hash in itertools.islice(full_block_hashes, max_length // block_size):
            cached_block = block_pool.get_cached_block(block_hash, kv_cache_group_ids)
            if not cached_block:
                break
            for computed, cached in zip(computed_blocks, cached_block):
                computed.append(cached)
        hit_length = len(computed_blocks[0]) * block_size

Line 793 checks only max_length // block_size hashes. Line 794 asks the pool for a block per group. get_cached_block() builds the key from the hash and the group id. It returns the first block with that key, or None (vllm/v1/core/block_pool.py:198-224). A model with several groups needs one hit length that is valid for every group; The hybrid KV cache: specs, groups and one manager per layer type explains that search.

Step 6: adopting the hit#

The scheduler passes the hit blocks to allocate_slots(). The count check includes free hit blocks, because touch() will take them out of the free queue (The KV cache manager: blocks, the block pool and memory sizing, Step 3). If the blocks fit, allocate_slots() adds the hit blocks first and allocates the new blocks after that (vllm/v1/core/kv_cache_manager.py:581-600). The coordinator adds the hits in two phases. First it touches the hit blocks of every group, then it allocates blocks for external tokens of a KV connector. The comment names the bug that this order prevents:

        # Two-phase allocation (issue #33775): first touch every group's local
        # cache-hit blocks, then allocate external blocks for every group. This
        # ensures an earlier group's external `get_new_blocks` cannot evict a
        # later group's not-yet-touched cache-hit blocks.
        for i, manager in enumerate(self.single_type_managers):
            manager.add_local_computed_blocks(
                request_id,
                new_computed_blocks[i],
                num_local_computed_tokens,
                num_external_computed_tokens,
            )

        # Touch the computed blocks to make sure they won't be evicted.
        if self.enable_caching:
            self.block_pool.touch(new_computed_blocks)
        else:
            assert not any(new_computed_blocks), (
                "Computed blocks should be empty when prefix caching is disabled"
            )

        # Skip blocks are padded with null blocks.
        req_blocks.extend([self._null_block] * num_skipped_blocks)
        # Add the remaining computed blocks.
        req_blocks.extend(new_computed_blocks)
        # All cached hits (including skipped nulls) are already cached; mark
        # them so cache_blocks() will not try to re-cache blocks that already
        # have a block_hash set.
        self.num_cached_block[request_id] = len(req_blocks)

Line 305 is the touch. Line 318 records that the hit blocks are already cached, so the caching step does not hash them again. Line 312 puts null blocks where a sliding window has skipped blocks (see The hybrid KV cache: specs, groups and one manager per layer type).

Step 7: caching blocks that become full#

At the end of allocate_slots(), the manager hashes every block that will be full after this step:

        # P/D: delay caching blocks if we have to recv from
        # remote. Update state for locally cached blocks.
        if delay_cache_blocks:
            return self.create_kv_cache_blocks(new_blocks)

        # NOTE(woosuk): We want to commit (cache) up to num_local_computed_tokens
        # + num_external_computed_tokens + num_new_tokens, but must exclude
        # "non-committable" tokens (e.g., draft tokens that could be rejected).
        # Therefore, we cap the number at `request.num_tokens`, ensuring only
        # "finalized" tokens are cached.
        num_tokens_to_cache = min(
            total_computed_tokens + num_new_tokens,
            request.num_tokens,
        )
        self.coordinator.cache_blocks(request, num_tokens_to_cache)

1. Lines 604-605. With an asynchronous load from a KV connector, the data is not there yet, so caching waits. The scheduler caches these blocks when the transfer is done (vllm/v1/core/sched/scheduler.py:3254-3257).
2. Lines 612-616. Otherwise it caches up to computed + new tokens, but not past request.num_tokens. So draft tokens never get a hash. The hash is set when the step is scheduled, before the GPU computes the KV values.

The default AsyncScheduler plans the next step before the sampled token is known. That token is a placeholder and not part of num_tokens yet, so a block that it fills is not hashed at that allocation. The block gets its hash in a later allocate_slots(), after the token arrives. When tokens arrive, the async scheduler also caches up to num_computed_tokens − num_output_placeholders tokens (vllm/v1/core/sched/async_scheduler.py:75-79).

The single-type manager skips the blocks that already have a hash. Then it calls the pool for the new full blocks only (vllm/v1/core/single_type_kv_cache_manager.py:531-541):

        if not self.kv_cache_spec.prefix_cacheable:
            return
        num_cached_blocks = self.num_cached_block.get(request.request_id, 0)
        num_full_blocks = num_tokens // self.block_size

        if num_cached_blocks >= num_full_blocks:
            return

        for i, blk in enumerate(new_full_blocks):
            # Some blocks may be null or masked out when enabling sparse attention
            # like sliding window attention, or Mamba models with prefix-caching
            # in align mode. We skip null blocks here.
            if blk.is_null or (block_mask is not None and not block_mask[i]):
                continue
            block_hash = new_block_hashes[i]
            num_hash_tokens = (num_cached_blocks + i + 1) * block_size

            # Update and added the full block to the cache.
            block_hash_with_group_id = make_block_hash_with_group_id(
                block_hash, kv_cache_group_id
            )
            if blk.block_hash is not None:
                # The only valid case where a "new full block" already has a
                # hash is partial->full promotion of the same cache block.
                assert (
                    blk.block_hash_num_tokens is not None
                    and blk.block_hash_num_tokens < num_hash_tokens
                )
                removed_hashes = self._remove_cached_block_hashes(blk)
                self._emit_block_removed_events(removed_hashes)
            self._insert_block_hash(
                block_hash_with_group_id,
                blk,
                num_tokens=num_hash_tokens,
            )

Lines 277-278 skip null blocks and blocks that a mask leaves out (sliding-window and Mamba groups). Lines 295-299 set the hash on the block and insert it into the map. The map does not remove duplicates: two blocks with the same hash both stay, so the block ids in running block tables never change (vllm/v1/core/block_pool.py:48-52).

Step 8: free, eviction and reset#

The KV cache manager: blocks, the block pool and memory sizing (Step 5) and the basics deep dive "Free blocks, LRU order and eviction" explain the free order and the eviction. In short, a request frees its blocks in reverse order. At ref_cnt 0, an unhashed block goes to the front and a hashed block to the back (vllm/v1/core/block_pool.py:795-807). get_new_blocks() pops the front and evicts the old hash (vllm/v1/core/block_pool.py:729-750).

A reset drops every hash at once. The API server has the route only in dev mode (vllm/entrypoints/launchers/api_server/routers.py:46-49):

@router.post("/reset_prefix_cache")
async def reset_prefix_cache(
    raw_request: Request,
    reset_running_requests: bool = Query(default=False),
    reset_external: bool = Query(default=False),
):
    """Reset the local prefix cache.

    Optionally, if the query parameter `reset_external=true`
    also resets the external (connector-managed) prefix cache.

    Returns `{"success": bool}`. The reset fails (`success=false`) while
    blocks are still held, e.g. by running requests or in-flight async KV
    offload transfers; callers may retry.

    Example:
       POST /reset_prefix_cache?reset_external=true

    """
    logger.info("Resetting prefix cache...")

    success = await engine_client(raw_request).reset_prefix_cache(
        reset_running_requests, reset_external
    )
    return JSONResponse(content={"success": bool(success)})

Scheduler.reset_prefix_cache() can first preempt every running request (reset_running_requests=true), so that all blocks are free. Without it, the reset fails while any block is in use (vllm/v1/core/sched/scheduler.py:2950-2975, vllm/v1/core/block_pool.py:841-848). reset_external=true also resets the cache of the KV connector (vllm/v1/core/sched/scheduler.py:2977-2978). The docstring of the pool names two uses: RLHF after a weight update, and benchmarks (vllm/v1/core/block_pool.py:831-835).

Step 9: external hits from a KV connector#

A KV connector can hold KV blocks outside this engine: in CPU memory, on disk, or in another vLLM instance. After the local lookup, the scheduler asks the connector how many tokens after the local hit it can load (vllm/v1/core/sched/scheduler.py:1026-1039). A request that skips the prefix cache gets 0 external tokens too (vllm/v1/core/sched/scheduler.py:1040-1041). The external tokens are added to the computed tokens, and an asynchronous load puts the request into WAITING_FOR_REMOTE_KVS (vllm/v1/core/sched/scheduler.py:1101-1104, vllm/v1/core/sched/scheduler.py:1384-1393). The connector uses the same block hashes, which is one reason why the hasher also runs when only a connector is on (Step 1).

Step 10: KV events#

With KV events on, the pool queues an event for each change of the prefix cache. Turn them on with --kv-events-config (vllm/engine/arg_utils.py:1873):

vllm serve Qwen/Qwen3-0.6B \
  --kv-events-config '{"enable_kv_cache_events": true, "publisher": "zmq", "topic": "kv-events"}'

@config
class KVEventsConfig:
    """Configuration for KV event publishing."""

    enable_kv_cache_events: bool = False
    """If True, enable KV cache events for tracking block storage and removal.
    Events can be published externally by zmq using the event publisher config.
    """

    publisher: Literal["null", "zmq"] = None  # type: ignore[assignment]
    """The publisher to use for publishing kv events. Can be "null", "zmq".
    """

    endpoint: str = "tcp://*:5557"
    """The zmq endpoint to use for publishing kv events.
    """

    replay_endpoint: str | None = None
    """The zmq endpoint to use for replaying kv events.
    """

    buffer_steps: int = 10_000
    """The number of steps to cache for replay endpoint. Will only save
    events from the last N steps for the replay endpoint.
    """

    hwm: int = 100_000
    """The zmq high water mark for the event publisher. After queueing N events,
    events will start dropping if the consumer is not keeping up.
    """

    max_queue_size: int = 100_000
    """The maximum number of events to queue while waiting for publishing.
    """

    topic: str = ""
    """The topic to use for the event publisher. Consumers can subscribe to
    this topic to receive events.
    """

    def __post_init__(self):
        if self.publisher is None:
            self.publisher = "zmq" if self.enable_kv_cache_events else "null"

Three event types exist (vllm/distributed/kv_events.py:56-142):

The KV cache manager adds the spec kind and the window size of the group to each BlockStored (vllm/v1/core/kv_cache_manager.py:703-720). At the end of update_from_output(), the scheduler takes the events of the pool and of the connector, and publishes one KVEventBatch per step (vllm/v1/core/sched/scheduler.py:2469-2484). The ZMQ publisher runs its own thread. It sends three frames per batch: the topic, an 8-byte sequence number and the msgpack payload (vllm/distributed/kv_events.py:472-507). With replay_endpoint, a subscriber that missed a sequence number can ask for the batches from that number on. The publisher keeps the last buffer_steps batches for this (vllm/distributed/kv_events.py:306-329). Each data-parallel rank adds its rank to the port (vllm/distributed/kv_events.py:530-563).

What an external router can do with them. A router in front of several vLLM instances can subscribe to every instance. From BlockStored and BlockRemoved it keeps, per instance, the set of block hashes that the instance holds. For a new request, the router tokenizes the prompt and computes the same hash chain. Then it sends the request to the instance with the longest run of known hashes.

This works only when the router computes the same digests: the same block size, algorithm, extra keys and NONE_HASH seed. By default, the events carry only the low 64 bits of each digest as an integer. The router must convert its digests the same way (vllm/v1/core/kv_cache_utils.py:82-85). The default sha256 with its fixed seed allows it; xxhash without PYTHONHASHSEED does not (Step 3). sha256 hashes a Python pickle, so a router in another language needs sha256_cbor. The repository has a small subscriber that prints the events (examples/features/kv_events/kv_events_subscriber.py:72-120).

Step 11: the metrics#

vllm:prefix_cache_queries and vllm:prefix_cache_hits count tokens; the basics page shows the PromQL (Prefix caching: the idea). Three details are new here:

• The scheduler records a lookup only when the request is admitted (vllm/v1/core/sched/scheduler.py:1378-1382). A lookup for a request that waits is not counted.
• A request that was preempted before is counted in separate preempted_* fields, not in queries and hits (vllm/v1/metrics/stats.py:138-155). No logger exports these fields, so the Prometheus counters show only first admissions.
• The log line "Prefix cache hit rate" is a moving value over about the last 1,000 requests, not a total (vllm/v1/metrics/stats.py:39-57, vllm/v1/metrics/loggers.py:321-323).

With a KV connector, vllm:external_prefix_cache_queries and vllm:external_prefix_cache_hits count the external lookup (vllm/v1/metrics/loggers.py:685-707). Its queries are the tokens after the local hit (vllm/v1/core/sched/scheduler.py:1096-1099). vllm:prompt_tokens_cached_by_source splits the cached prompt tokens by tier: device, host, disk, p2p or external_unspecified (vllm/v1/metrics/loggers.py:781-796). Local hits count as device (vllm/v1/metrics/loggers.py:1174-1177).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

        block_size = kv_cache_spec.block_size
        if dcp_world_size > 1:
            # DCP shards each block's KV across ranks; hashes must be viewed at
            # the sharded block size.
            block_size *= dcp_world_size
        block_hashes = resolve_block_hashes(
            block_hashes,
            block_pool.hash_block_size,
            block_size,
            supports_fine_grained_hash_lookup=cls.supports_fine_grained_hash_lookup,
            alignment_tokens=alignment_tokens,
        )

        # Fine-grained mode (alignment_tokens == hash_block_size <
        # block_size): resolve_block_hashes kept the raw hash-granularity
        # list so interior boundaries can be probed.
        fine_grained = (
            alignment_tokens < block_size and block_size % alignment_tokens == 0
        )
        if fine_grained:
            # list or lazy BlobBlockHashes view
            assert isinstance(block_hashes, Sequence)
            full_block_hashes: BlockHashList = BlockHashListWithBlockSize(
                block_hashes, alignment_tokens, block_size
            )
        else:
            full_block_hashes = block_hashes

        computed_blocks: tuple[list[KVCacheBlock], ...] = tuple(
            [] for _ in range(len(kv_cache_group_ids))
        )
        # Phase 1: longest run of cached full blocks from the start. A missing
        # block implies every later block misses too (chained hashes).
        for block_hash in itertools.islice(full_block_hashes, max_length // block_size):
            cached_block = block_pool.get_cached_block(block_hash, kv_cache_group_ids)
            if not cached_block:
                break
            for computed, cached in zip(computed_blocks, cached_block):
                computed.append(cached)
        hit_length = len(computed_blocks[0]) * block_size

        # Phase 2 (fine-grained only): extend into the first non-full block by
        # probing its interior hash boundaries high-to-low (longest hit first).
        if fine_grained:
            # list or lazy BlobBlockHashes view
            assert isinstance(block_hashes, Sequence)
            scale_factor = block_size // alignment_tokens
            first_partial_idx = len(computed_blocks[0]) * scale_factor
            max_partial_idx = min(
                first_partial_idx + scale_factor - 1,
                max_length // alignment_tokens,
                len(block_hashes),
            )
            for fine_idx in range(max_partial_idx - 1, first_partial_idx - 1, -1):
                cached_tail = block_pool.get_cached_block(
                    block_hashes[fine_idx], kv_cache_group_ids
                )
                if not cached_tail:
                    continue
                for computed, cached in zip(computed_blocks, cached_tail):
                    computed.append(cached)
                hit_length = (fine_idx + 1) * alignment_tokens
                break

        # Eagle needs the tokens right before the generation point recomputed:
        # drop one hash unit when fine-grained (the tail block's KV is
        # append-only, so it still covers the reduced length), else one cache
        # block.
        if drop_eagle_block and hit_length > 0:
            hit_length -= min(alignment_tokens, block_size)
        # Round down to the alignment; a no-op when fine-grained (hits land on
        # hash boundaries by construction) and when alignment_tokens ==
        # block_size. Then trim blocks past the new tail.
        hit_length -= hit_length % alignment_tokens
        num_blocks = cdiv(hit_length, block_size)
        for computed in computed_blocks:
            del computed[num_blocks:]
        return computed_blocks, hit_length

What can go wrong#

• The router sends requests to the wrong instance. The router hashes with another algorithm, block size or seed than the engines. Its hashes never match the events. Use the default sha256, or set PYTHONHASHSEED on every instance, and convert digests the same way as VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES (vllm/v1/core/kv_cache_utils.py:82-85).
• Prompt logprobs give no cache benefit. The request skips the lookup and the external lookup (vllm/sampling_params.py:625-632, vllm/v1/core/sched/scheduler.py:1040-1041).
• The reset route returns 404. The dev routes exist only with VLLM_SERVER_DEV_MODE (vllm/entrypoints/launchers/api_server/routers.py:46-49). A reset under load returns false unless you also pass reset_running_requests=true.
• The hit rate in the log differs from Prometheus. The log uses a window of recent requests; the counters are totals since start-up (vllm/v1/metrics/stats.py:47-57).
• An identical prompt still computes a whole block. The cap is num_tokens − 1, and hits are whole blocks. A prompt of 4,096 tokens hits at most 4,080 tokens with block size 16.

Why it matters. The prefix cache uses no extra memory: it has no size limit and no timer, only the order of the free queue. The metrics, a prefix-aware router and a reset after a weight update all depend on one fact. It is when a block gets its hash, and when it loses it.