v1/core/kv_cache_manager.py: KVCacheManager and KVCacheBlocks

Code version: vLLM @ 3e44c0d · 2026-10-22

Purpose#

vllm/v1/core/kv_cache_manager.py gives the scheduler one small API for KV blocks, and hides the coordinator, the per-group managers and the block pool behind it.

The page The KV cache manager: blocks, the block pool and memory sizing explains the ideas with a figure. This page reads the file.

Where it sits#

Who calls it#

• Scheduler.__init__ builds it (vllm/v1/core/sched/scheduler.py:329-347).
• The waiting pass calls get_computed_blocks through _get_local_prefix_cache_hit (vllm/v1/core/sched/scheduler.py:589-599), and allocate_slots with the hit (vllm/v1/core/sched/scheduler.py:1320-1337).
• The running pass calls allocate_slots with new tokens only (vllm/v1/core/sched/scheduler.py:834-838).
• _free_request_blocks calls free or pop_blocks_for_free (vllm/v1/core/sched/scheduler.py:2866-2875).
• AsyncScheduler calls cache_blocks when sampled tokens arrive (vllm/v1/core/sched/async_scheduler.py:75-79).
• make_stats and get_kv_cache_usage read usage and make_prefix_cache_stats (vllm/v1/core/sched/scheduler.py:2709-2711, vllm/v1/core/sched/scheduler.py:3025).
• A KV connector gets the manager after construction (vllm/v1/core/sched/scheduler.py:359-361).

What it calls#

• get_kv_cache_coordinator (vllm/v1/core/kv_cache_manager.py:167-180), which picks KVCacheCoordinatorNoPrefixCache, UnitaryKVCacheCoordinator or HybridKVCacheCoordinator (vllm/v1/core/kv_cache_coordinator.py:995-1038).
• The coordinator methods find_longest_cache_hit, get_num_blocks_to_allocate, allocate_new_computed_blocks, allocate_new_blocks, cache_blocks, remove_skipped_blocks, free (vllm/v1/core/kv_cache_coordinator.py:190-473).
• BlockPool directly for usage, events and evictions (vllm/v1/core/kv_cache_manager.py:197, vllm/v1/core/kv_cache_manager.py:234).

Key classes and functions#

Walk through the main methods#

KVCacheBlocks: the result type#

@dataclass
class KVCacheBlocks:
    """The allocation result of KVCacheManager, work as the interface between
    Scheduler and KVCacheManager, to hide KVCacheManager's internal data
    structure from the Scheduler.
    """

    blocks: tuple[Sequence[KVCacheBlock], ...]
    """
    `blocks[i][j]` refers to the i-th kv_cache_group
    and the j-th block of tokens.We don't use block of
    tokens as the outer dimension because it assumes all
    kv_cache_groups have the same number of blocks, which is true for now but
    will be broken if we want to give different block_size to different
    kv_cache_groups in the future.

    Each single type KVCacheBlocks could be represented as:
    - list[KVCacheBlock] for more than one KVCacheBlock
    - an empty tuple for requests without KVCacheBlock
      (a precomputed KVCacheBlocks is in KVCacheManager to avoid GC overhead)
    """

1. Line 41. A tuple with one sequence per KV cache group. A model with only full attention has one group.
2. Lines 43-48. The group is the outer level, because groups may get different block sizes.
3. Lines 50-53. An empty result is a tuple of empty tuples, made once and shared, to save garbage collection work.

get_computed_blocks: the prefix-cache hit#

        if not self.prefix_cache_lookup_enabled(request):
            return self.empty_kv_cache_blocks, 0, 0

        # NOTE: When all tokens hit the cache, we must recompute the last token
        # to obtain logits. Thus, set max_cache_hit_length to prompt_length - 1.
        # This can trigger recomputation of an entire block, rather than just
        # the single last token, because allocate_slots() requires
        # num_computed_tokens to be block-size aligned. Removing this limitation
        # could slightly improve performance in the future.
        max_cache_hit_length = request.num_tokens - 1
        computed_blocks, num_new_computed_tokens, num_uncached = (
            self.coordinator.find_longest_cache_hit(
                request.block_hashes, max_cache_hit_length
            )
        )

1. Lines 286-287. No lookup when prefix caching is off, or when the request skips the cache read (for example a request with prompt logprobs) (vllm/v1/core/kv_cache_manager.py:249-251).
2. Line 295. At most num_tokens - 1 tokens: the last prompt token must run to produce logits. The comment says that this can cost a whole block of recompute.
3. Lines 296-300. The coordinator searches the hash chain. How the search works is on Prefix caching in the engine core: hashes, lookup, caching, eviction and KV events.

allocate_slots: count, then change#

The concept page reads this method in two parts (The KV cache manager: blocks, the block pool and memory sizing, Steps 3 and 4). The order of its stages, line by line:

1. Lines 474-482. ValueError for a call with nothing to do.
2. Lines 484-497. Take the new hit blocks, cut for groups that keep a longer hit.
3. Lines 501-507. num_local_computed_tokens and total_computed_tokens.
4. Lines 509-516. Watermark blocks, only for waiting or preempted requests while others run.
5. Lines 522-539. full_sequence_must_fit: the whole request must fit, or None.
6. Lines 541-544. num_tokens_need_slot, with the lookahead, capped at max_model_len.
7. Lines 555-559. Free blocks outside the attention window, on the processed tokens only.
8. Lines 561-579. Count the blocks, compare with the free blocks minus reservations, or None.
9. Lines 581-593. Adopt the hit blocks and allocate blocks for connector tokens.
10. Lines 595-600. Allocate the new blocks.
11. Lines 602-616. Cache full blocks, unless a KV load is still running.
12. Line 618. Return only the new blocks.

        num_blocks_to_allocate = self.coordinator.get_num_blocks_to_allocate(
            request_id=request.request_id,
            num_tokens=num_tokens_need_slot,
            new_computed_blocks=new_computed_block_list,
            num_encoder_tokens=num_encoder_tokens,
            total_computed_tokens=num_local_computed_tokens
            + num_external_computed_tokens,
            num_local_computed_tokens=num_local_computed_tokens,
            num_tokens_main_model=num_tokens_main_model,
            prefill_end=prefill_end,
        )

        # Keep `reserved_blocks` free for other in-flight sequences, and an
        # additional watermark of headroom for waiting/preempted admissions.
        available_blocks = self.block_pool.get_num_free_blocks() - reserved_blocks
        required_blocks = num_blocks_to_allocate + watermark_blocks
        if required_blocks > available_blocks:
            # Cannot allocate new blocks
            return None

The check uses reserved_blocks, which the scheduler sets only for asynchronous KV loads (vllm/v1/core/sched/scheduler.py:1305-1315).

free and usage#

    def free(self, request: Request) -> None:
        """Free the blocks allocated for the request.
        We free the blocks in reverse order so that the tail blocks are evicted
        first when caching is enabled.

        Args:
            request: The request to free the blocks.

        """
        self.coordinator.free(request.request_id)

The docstring describes the reverse order, but the reverse happens in the single-type manager (vllm/v1/core/single_type_kv_cache_manager.py:597-598). usage forwards to BlockPool.get_usage: 1 - free / (num_gpu_blocks - 1) (vllm/v1/core/block_pool.py:881-892).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

Related concepts#

• The KV cache manager: blocks, the block pool and memory sizing: the ideas, memory sizing and the usage metric.
• Prefix caching in the engine core: hashes, lookup, caching, eviction and KV events: the hash chain and the hit search.
• The hybrid KV cache: specs, groups and one manager per layer type: several KV cache groups.
• v1/core/block_pool.py: BlockPool and the prefix-cache map: the pool under this manager.
• v1/core/sched/scheduler.py: the Scheduler: the caller.