The KV cache manager: blocks, the block pool and memory sizing
Code version: vLLM @ 3e44c0d · 2026-10-22
In one sentence
The engine core keeps a ledger of KV blocks: who holds them, which are cached and which are free. The GPU tensors live in the workers.
The idea#
The KV cache is split into fixed blocks, and each request has a table of its blocks (The KV cache, Paged KV cache). In vLLM the blocks are 16 tokens by default. This page explains who manages them in vLLM, and how.
The manager runs in the EngineCore process, next to the scheduler. It never touches GPU memory. It keeps one small Python object per block, a KVCacheBlock, with an id, a reference count and an optional hash. The real KV tensors live in the workers, one copy per GPU. The scheduler decides; the workers only receive block ids in the SchedulerOutput and write into those blocks.
The manager has four layers. KVCacheManager is the API that the scheduler calls. A KVCacheCoordinator spreads each call over the KV cache groups. A SingleTypeKVCacheManager per group knows the rules of one attention type, for example full attention or a sliding window. One block pool owns all blocks and the free block queue. A plain decoder model such as Llama has one group, so the coordinator is a thin wrapper.
The pool shares blocks through reference counts. Two requests with the same prefix can point to the same full block. The block returns to the free queue only when its count reaches 0. A free block keeps its hash, so it can still serve a prefix-cache hit until the pool reuses it (Prefix caching: the idea). How many blocks exist is decided once, at start-up, from a memory profile of the GPU.
An everyday comparison. Think of a warehouse office. The office keeps a ledger of shelf numbers: who rents which shelf, which shelves hold a well-known kit, and which are empty. The workers in the warehouse only get shelf numbers and do the lifting. The comparison stops at the empty shelves. An empty shelf in vLLM still holds its old contents. A new tenant who needs the same kit can take it as it is.
The picture#
Read panel 1 from left to right and down: each layer owns the next. Panel 2 is the call that Tour 1 makes at stop 10 (Tour 1, stop 10). Panel 3 shows why cached blocks survive a free. They go to the end of the queue, and the pool takes new blocks from the front.
In the code#
Step 1: who builds what#
The scheduler builds the manager in its __init__, with the KV cache configuration from start-up and the prefix caching flag:
self.kv_cache_manager = KVCacheManager(
kv_cache_config=kv_cache_config,
max_model_len=self.max_model_len,
max_in_flight_tokens=vllm_config.max_in_flight_tokens,
enable_caching=self.cache_config.enable_prefix_caching,
use_eagle=self.use_eagle_block_drop,
num_prefill_lookahead=self.num_prefill_lookahead,
log_stats=self.log_stats,
enable_kv_cache_events=self.enable_kv_cache_events,
dcp_world_size=self.dcp_world_size,
pcp_world_size=1,
scheduler_block_size=self.block_size,
hash_block_size=hash_block_size,
metrics_collector=self.kv_metrics_collector,
watermark=self.scheduler_config.watermark,
enable_mamba_shared_prefix_checkpoint=(
self.cache_config.enable_mamba_shared_prefix_checkpoint
),
)The manager builds the coordinator (vllm/v1/core/kv_cache_manager.py:167-180). The factory picks one of three classes: no prefix caching, exactly one KV cache group, or several groups:
if not enable_caching:
return KVCacheCoordinatorNoPrefixCache(
kv_cache_config,
max_model_len,
max_in_flight_tokens,
use_eagle,
enable_kv_cache_events,
dcp_world_size=dcp_world_size,
pcp_world_size=pcp_world_size,
scheduler_block_size=scheduler_block_size,
hash_block_size=hash_block_size,
metrics_collector=metrics_collector,
num_prefill_lookahead=num_prefill_lookahead,
)
if len(kv_cache_config.kv_cache_groups) == 1:
return UnitaryKVCacheCoordinator(
kv_cache_config,
max_model_len,
max_in_flight_tokens,
use_eagle,
enable_caching,
enable_kv_cache_events,
dcp_world_size=dcp_world_size,
pcp_world_size=pcp_world_size,
scheduler_block_size=scheduler_block_size,
hash_block_size=hash_block_size,
metrics_collector=metrics_collector,
num_prefill_lookahead=num_prefill_lookahead,
)
return HybridKVCacheCoordinator(Every coordinator makes one BlockPool with kv_cache_config.num_blocks blocks, and one single-type manager per KV cache group, all on the same pool. The only exception is the host group of HiSparse, which replaces the pool of its manager with a private host pool (vllm/v1/core/single_type_kv_cache_manager.py:2231-2247). For this reason a block remembers its pool, and free_blocks sends a block of another pool back to that pool (vllm/v1/core/block_pool.py:792-794, vllm/v1/core/block_pool.py:808-809).
self.block_pool = BlockPool(
num_gpu_blocks=kv_cache_config.num_blocks,
enable_caching=enable_caching,
hash_block_size=hash_block_size,
enable_kv_cache_events=enable_kv_cache_events,
metrics_collector=metrics_collector,
) self.single_type_managers = tuple(
get_manager_for_kv_cache_spec(
kv_cache_spec=kv_cache_group.kv_cache_spec,
max_in_flight_tokens=max_in_flight_tokens,
max_model_len=max_model_len,
block_pool=self.block_pool,
role=kv_cache_group.role,
enable_caching=enable_caching,
kv_cache_group_id=i,
dcp_world_size=dcp_world_size,
pcp_world_size=pcp_world_size,
scheduler_block_size=self.scheduler_block_size,
needs_kv_cache_zeroing=self.kv_cache_config.needs_kv_cache_zeroing,
)
for i, kv_cache_group in enumerate(self.kv_cache_config.kv_cache_groups)
)A registry maps each spec class to a manager class: FullAttentionSpec and MLAAttentionSpec to FullAttentionManager, SlidingWindowSpec to SlidingWindowManager, MambaSpec to MambaManager, and more (vllm/v1/core/single_type_kv_cache_manager.py:2645-2715). A platform can register its own specs (vllm/v1/core/single_type_kv_cache_manager.py:2717-2719). Models with several groups (for example sliding window plus full attention) are the topic of The hybrid KV cache: specs, groups and one manager per layer type.
Step 2: KVCacheBlock and the BlockPool#
@dataclass(slots=True)
class KVCacheBlock:
"""KV-cache block metadata."""
# Block ID, ranging from 0 to num_gpu_blocks - 1.
block_id: int
# Reference count.
ref_cnt: int = 0
# Deferred frees may contain blocks from multiple pools.
pool: "BlockPool | None" = field(
default=None, repr=False, compare=False, kw_only=True
)
# The hash key (block hash + group id) of the block, only available
# when the block is full and cached.
_block_hash: BlockHashWithGroupId | None = None
# Number of prefix tokens covered by _block_hash. For full blocks this is
# the full block boundary; partial entries can end inside a cache block.
_block_hash_num_tokens: int | None = None
# Used to construct a doubly linked list for free blocks.
# These two attributes should only be manipulated by FreeKVCacheBlockQueue.
prev_free_block: "KVCacheBlock | None" = None
next_free_block: "KVCacheBlock | None" = None
# Whether the block is a null block that should never be cached.
is_null: bool = FalseA block has an id from 0 to num_gpu_blocks - 1 and a reference count. It has a hash only when it is full and cached, and two links for the free queue. The pool makes all blocks at once, puts them all in the free queue, and takes out block 0:
# All kv-cache blocks.
self.blocks: list[KVCacheBlock] = [
KVCacheBlock(idx, pool=self) for idx in range(num_gpu_blocks)
]
# Free block queue that constructs and manipulates a doubly linked
# list of free blocks (including eviction candidates when caching is
# enabled).
self.free_block_queue = FreeKVCacheBlockQueue(self.blocks)
# Cache for block lookup
self.cached_block_hash_to_block: BlockHashToBlockMap = BlockHashToBlockMap()
self.cached_block_hashes_by_block: dict[int, set[BlockHashWithGroupId]] = {}
# To represent a placeholder block with block_id=0.
# The ref_cnt of null_block is not maintained, needs special care to
# avoid freeing it.
self.null_block = self.free_block_queue.popleft()
self.null_block.is_null = TrueBlock 0 becomes the null block. No request gets it as a real block. Managers put it in block tables at places that hold no data, for example blocks that a sliding window has left (vllm/v1/core/single_type_kv_cache_manager.py:311-312). So a pool of N blocks has N − 1 usable blocks.
The free queue is a doubly linked list through the blocks themselves, so it can remove a block from the middle in O(1) (vllm/v1/core/kv_cache_utils.py:258-279). Its order is the LRU order of prefix caching, which Prefix caching: the idea explains in its deep dive "Free blocks, LRU order and eviction".
Step 3: allocate_slots, the count#
The scheduler calls allocate_slots for every scheduled request. A running request passes only num_new_tokens. A new request also passes its prefix-cache hit. The method first counts, and changes nothing until it knows that the blocks fit:
# The number of computed tokens is the number of computed tokens plus
# the new prefix caching hits
num_local_computed_tokens = (
request.num_computed_tokens + num_new_computed_tokens
)
total_computed_tokens = min(
num_local_computed_tokens + num_external_computed_tokens,
self.max_model_len,
)
watermark_blocks = 0
# The watermark is applied to waiting/preempted requests only, and only
# when there's at least one request already scheduled.
if has_scheduled_reqs and request.status in (
RequestStatus.WAITING,
RequestStatus.PREEMPTED,
):
watermark_blocks = self.watermark_blocks
# Matches the scheduler's own prefill boundary: `num_tokens - 1`
# extends it to resumed requests replaying their output tokens.
prefill_end = max(request.num_prompt_tokens, request.num_tokens - 1)
if full_sequence_must_fit:
# First check and fail if the full request sequence won't fit.
full_num_tokens = min(request.num_tokens, self.max_model_len)
num_blocks_to_allocate = self.coordinator.get_num_blocks_to_allocate(
request_id=request.request_id,
num_tokens=full_num_tokens,
new_computed_blocks=new_computed_block_list,
num_encoder_tokens=num_encoder_tokens,
total_computed_tokens=total_computed_tokens,
num_local_computed_tokens=num_local_computed_tokens,
num_tokens_main_model=full_num_tokens,
apply_admission_cap=True,
prefill_end=prefill_end,
)
required_blocks = num_blocks_to_allocate + watermark_blocks
if required_blocks > self.block_pool.get_num_free_blocks():
return None
num_tokens_main_model = total_computed_tokens + num_new_tokens
num_tokens_need_slot = min(
num_tokens_main_model + num_lookahead_tokens, self.max_model_len
)- Lines 501-507. The tokens that already have KV values: the computed tokens, plus the new prefix-cache hit, plus tokens from a KV connector.
- Lines 512-516. A watermark (default 0) applies only to waiting and preempted requests, and only when other requests run.
- Lines 522-539. The admission gate for new requests: all their tokens (at most
max_model_len) must fit, not only the first chunk (scheduler_reserve_full_isl, default True). Sliding-window and chunked-local groups cap this count, because they free blocks as the request moves on (vllm/v1/core/single_type_kv_cache_manager.py:215-227). The scheduler passes the flag only in the waiting pass. - Lines 541-544. The slots needed: computed plus new tokens, plus
num_lookahead_tokensfor speculative decoding, at mostmax_model_len.
The single-type manager turns slots into blocks. For a running request, it is the difference between the blocks needed and the blocks held:
num_required_blocks = cdiv(num_tokens, self.block_size)
if apply_admission_cap and self.max_admission_blocks_per_request is not None:
# Recycling-aware specs (SWA, chunked-local) cap the per-request
# reservation here so admission matches the startup pool sizer
# (`SlidingWindowSpec.max_admission_blocks_per_request` / its
# chunked-local counterpart). `remove_skipped_blocks` runs from
# `allocate_slots` before each chunk's `get_num_blocks_to_allocate`,
# so per-request peak real-held blocks <= this cap, which keeps
# `sum(reservations) <= pool` <=> `sum(peak_real_held) <= pool`.
# Drift between the two would re-introduce the deadlock from
# issue #39734 or, worse, mid-prefill OOM.
num_required_blocks = min(
num_required_blocks, self.max_admission_blocks_per_request
)
num_req_blocks = len(self.req_to_blocks.get(request_id, ()))
if request_id in self.num_cached_block:
# Fast-path: a running request won't have any new prefix-cache hits.
assert len(new_computed_blocks) == 0
# NOTE: With speculative decoding, request's blocks may be allocated
# for draft tokens which are later rejected. In this case,
# num_required_blocks may be smaller than num_req_blocks.
return max(num_required_blocks - num_req_blocks, 0)For a new request, it also counts the prefix-cache hits that are free right now (ref_cnt == 0). The request will take them out of the free queue, so they use up free capacity too (vllm/v1/core/single_type_kv_cache_manager.py:256-266).
Step 4: allocate_slots, the changes#
# Free the blocks that are skipped during the attention computation
# (e.g., tokens outside the sliding window).
# We can do this even if we cannot schedule this request due to
# insufficient free blocks.
# Should call this function before allocating new blocks to reduce
# the number of evicted blocks.
# Free on the processed-token basis: in-flight steps' attention windows
# still read blocks below the optimistic boundary, and rejected spec
# tokens can roll it back.
self.coordinator.remove_skipped_blocks(
request.request_id,
max(0, total_computed_tokens - request.num_in_flight_tokens),
num_prompt_tokens=request.num_prompt_tokens,
)
num_blocks_to_allocate = self.coordinator.get_num_blocks_to_allocate(
request_id=request.request_id,
num_tokens=num_tokens_need_slot,
new_computed_blocks=new_computed_block_list,
num_encoder_tokens=num_encoder_tokens,
total_computed_tokens=num_local_computed_tokens
+ num_external_computed_tokens,
num_local_computed_tokens=num_local_computed_tokens,
num_tokens_main_model=num_tokens_main_model,
prefill_end=prefill_end,
)
# Keep `reserved_blocks` free for other in-flight sequences, and an
# additional watermark of headroom for waiting/preempted admissions.
available_blocks = self.block_pool.get_num_free_blocks() - reserved_blocks
required_blocks = num_blocks_to_allocate + watermark_blocks
if required_blocks > available_blocks:
# Cannot allocate new blocks
return None if (
new_computed_block_list is not self.empty_kv_cache_blocks.blocks
or num_external_computed_tokens > 0
):
# Append the new computed blocks to the request blocks until now to
# avoid the case where the new blocks cannot be allocated.
self.coordinator.allocate_new_computed_blocks(
request_id=request.request_id,
new_computed_blocks=new_computed_block_list,
num_local_computed_tokens=num_local_computed_tokens,
num_external_computed_tokens=num_external_computed_tokens,
skip_zeroing_group_ids=skip_zeroing_group_ids,
)
new_blocks = self.coordinator.allocate_new_blocks(
request.request_id,
num_tokens_need_slot,
num_tokens_main_model,
num_encoder_tokens,
)
# P/D: delay caching blocks if we have to recv from
# remote. Update state for locally cached blocks.
if delay_cache_blocks:
return self.create_kv_cache_blocks(new_blocks)
# NOTE(woosuk): We want to commit (cache) up to num_local_computed_tokens
# + num_external_computed_tokens + num_new_tokens, but must exclude
# "non-committable" tokens (e.g., draft tokens that could be rejected).
# Therefore, we cap the number at `request.num_tokens`, ensuring only
# "finalized" tokens are cached.
num_tokens_to_cache = min(
total_computed_tokens + num_new_tokens,
request.num_tokens,
)
self.coordinator.cache_blocks(request, num_tokens_to_cache)
return self.create_kv_cache_blocks(new_blocks)- Lines 555-559. Blocks that the attention no longer reads (outside a sliding window) go back first, so the pool has more free blocks. For full attention this does nothing.
- Lines 561-579. The count, then the check: not enough free blocks means
return None. Nothing has changed yet. The scheduler then preempts a request or stops its waiting pass. - Lines 581-593. A new request adopts its prefix-cache hit. The manager touches the hit blocks (
ref_cnt += 1, out of the free queue). Then it puts them at the start of the request's block table (vllm/v1/core/single_type_kv_cache_manager.py:303-318). - Lines 595-600. New blocks for the rest. The single-type manager takes them from the front of the free queue (
vllm/v1/core/single_type_kv_cache_manager.py:400-410). The pool removes an old hash from a reused block (this is the eviction) and setsref_cntto 1 (vllm/v1/core/block_pool.py:681-700). - Lines 604-605. With an asynchronous KV load, the blocks are not cached yet: the data is not there.
- Lines 612-616. The blocks that are now full get a hash, so later requests can hit them. Draft tokens are left out: the count stops at
request.num_tokens.
The return value holds only the new blocks of this call. For a new request, the scheduler reads all its blocks with get_blocks() instead (vllm/v1/core/sched/scheduler.py:1432-1434).
Step 5: free, in reverse order#
When a request finishes or is preempted, the scheduler calls free() (vllm/v1/core/sched/scheduler.py:2866-2872). Each single-type manager gives its blocks back, last block first:
def free(self, request_id: str) -> None:
"""Free the blocks for the request.
Args:
request_id: The request ID.
"""
# Free blocks in reverse order so that the tail blocks are freed first.
self.block_pool.free_blocks(reversed(self.pop_blocks_for_free(request_id)))BlockPool.free_blocks lowers each ref_cnt. A block that reaches 0 goes back into the free queue: at the front without a hash, at the end with a hash (vllm/v1/core/block_pool.py:776-807). A block that another request still uses stays out of the queue. Because the last block comes first, the tail of a cached prefix is evicted before its head. Prefix caching itself (hashes, lookup, eviction) is the topic of Prefix caching in the engine core: hashes, lookup, caching, eviction and KV events.
Step 6: how the block ids reach the worker#
The manager returns KVCacheBlocks, a tuple with one list per KV cache group. The scheduler turns it into ids:
groups = (
self.blocks
if group_ids is None
else tuple(self.blocks[group_id] for group_id in group_ids)
)
if allow_none and all(len(group) == 0 for group in groups):
return None
return tuple([blk.block_id for blk in group] for group in groups)A new request carries all its ids in NewRequestData.block_ids. A running request carries only the ids of new blocks in CachedRequestData.new_block_ids, or None when it got no new block (vllm/v1/core/sched/output.py:47, vllm/v1/core/sched/output.py:140-150, vllm/v1/core/sched/scheduler.py:1821-1823). Model Runner V2 writes them into its block tables on the GPU. It overwrites the row of a new request and appends to the row of a running one:
self.block_tables.append_block_ids(
req_index, new_req_data.block_ids, overwrite=True
) for req_id, num_computed_tokens, req_new_block_ids in zip(
reqs.req_ids, reqs.num_computed_tokens, reqs.new_block_ids
):
req_index = self.req_states.req_id_to_index[req_id]
num_computed_tokens_np[req_index] = num_computed_tokens
if req_new_block_ids is not None:
self.block_tables.append_block_ids(
req_index, req_new_block_ids, overwrite=False
)The block table is an int32 tensor [max_num_reqs, max_num_blocks] per KV cache group (vllm/v1/worker/gpu/block_table.py:65-71). The slot of a token is block_id × block_size + offset (vllm/v1/worker/gpu/block_table.py:409). A preempted request leaves the runner like a finished one (vllm/v1/worker/gpu/model_runner.py:1178-1185). When it resumes, it comes back as a new request with a fresh list of block ids (vllm/v1/core/sched/scheduler.py:1505-1507). The page The model runner (Model Runner V2) explains the runner side.
Step 7: how many blocks, from the memory profile#
The number of blocks is fixed once, in EngineCore._initialize_kv_caches. Each worker measures the memory that is left for the KV cache (vllm/v1/engine/core.py:297-301). The executor and worker pages explain the profile run and the formula (Executors and workers: from the engine core to the GPUs):
Then the engine core divides the bytes by the bytes of one block, over all layers of the group:
bytes_per_block = _pool_bytes_per_block_from_builder(
builder, vllm_config, projected_groups
)
num_blocks = (
available_memory_one_worker // bytes_per_block if projected_groups else 1
)The engine core gets bytes_per_block from a probe layout with one block. For a plain model, that is the sum of the page sizes of all layers. A helper computes the same divisor directly (vllm/v1/core/kv_cache_planning.py:251-262):
bytes_per_block = max(
sum(
_get_per_layer_spec(group, layer_name).page_size_bytes
for layer_name in group.layer_names
)
for group in kv_cache_groups
)The page of one layer is num_kv_heads × block_size × (head_size + head_size_v) × dtype bytes (vllm/v1/kv_cache_interface.py:571-579, vllm/v1/kv_cache_interface.py:254-255). With several workers (TP), every worker gets the smallest block count of all workers (vllm/v1/core/kv_cache_planning.py:1622-1636). The result goes into cache_config.num_gpu_blocks, which the scheduler reads (vllm/v1/engine/core.py:324-325, vllm/v1/core/sched/scheduler.py:207-208).
Two settings change the input of this division:
--gpu-memory-utilization, default 0.92: the share of the whole GPU that this vLLM instance may use, for weights, activations and KV cache together (vllm/config/cache.py:212-221).--kv-cache-memory-bytes: a fixed KV cache size per GPU. The worker then skips the memory measurement, andgpu_memory_utilizationhas no effect (vllm/config/cache.py:371-378,vllm/v1/worker/gpu_worker.py:653-679).
Step 8: what "KV cache usage" means#
The usage that vLLM reports is the share of blocks that are not in the free queue:
def get_usage(self) -> float:
"""Get the KV cache usage.
Returns:
The KV cache usage (between 0.0 and 1.0).
"""
# Subtract 1 to account for null block.
total_gpu_blocks = self.num_gpu_blocks - 1
if not total_gpu_blocks:
return 0
return 1.0 - (self.get_num_free_blocks() / total_gpu_blocks)When stats are on, the scheduler puts this value into its stats every step (vllm/v1/core/sched/scheduler.py:3023-3024, vllm/v1/core/sched/scheduler.py:3050). The front end shows it as the gauge vllm:kv_cache_usage_perc, with values from 0 to 1 in spite of its name (vllm/v1/metrics/loggers.py:619-623). The log line shows "GPU KV cache usage: x%" (vllm/v1/metrics/loggers.py:314-320).
Deep dive#
The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.
Deep diveThe block layout of allocate_slots
The docstring of allocate_slots draws the token ranges of one call (vllm/v1/core/kv_cache_manager.py:420-452):
| comp | new_comp | ext_comp | new | lookahead |
comp = request.num_computed_tokens (blocks already held)
new_comp = num_new_computed_tokens (local prefix-cache hit, this call)
ext_comp = num_external_computed_tokens (KV connector hit)
new = num_new_tokens (to compute; may include draft tokens)
lookahead = num_lookahead_tokens (slots for the drafter)Allocated: everything after comp. Computed in this step: new and lookahead. Cached: up to min(comp + new_comp + ext_comp + new, request.num_tokens) (vllm/v1/core/kv_cache_manager.py:612-615).
The method raises ValueError when a call asks for 0 new tokens and adopts no computed tokens (vllm/v1/core/kv_cache_manager.py:474-482). With prefix-cache retention of some groups (Mamba, sliding window), the hit is first cut to the reused prefix (lines 484-497).
Deep diveWorked example: Llama-3.1-8B on one 80 GiB GPU
Llama-3.1-8B has 32 layers, 8 KV heads and a head size of 128; the KV cache is BF16 (2 bytes). These model values are from its Hugging Face config (inferred, not in the vLLM repo).
- One layer, one block. 8 heads × 16 tokens × (128 + 128) × 2 bytes = 65,536 bytes = 64 KiB (
vllm/v1/kv_cache_interface.py:571-579). - One block, all layers. 32 × 64 KiB = 2 MiB (
vllm/v1/core/kv_cache_planning.py:587-593). - Blocks. With 54.6 GiB available (figure above): 55,910 MiB / 2 MiB = 27,955 blocks (
vllm/v1/core/kv_cache_planning.py:1610-1612). Block 0 is the null block, so 27,954 blocks can hold tokens. - Tokens. 27,955 × 16 = 447,280 token slots.
- Concurrency. With
max_model_len131,072, one full-length request needs 8192 blocks. 27,955 / 8192 = 3.41. The start-up log prints "GPU KV cache size: 447,280 tokens, Maximum concurrency for 131,072 tokens per request: 3.41x" (vllm/v1/core/kv_cache_utils.py:1090-1108,vllm/v1/core/kv_cache_utils.py:1015-1047).
The capacity log counts all num_blocks, including the null block (derived from the lines above). The difference of one block does not matter in practice.
At runtime, a chat request with 2000 prompt tokens and 500 output tokens holds cdiv(2500, 16) = 157 blocks at its end. That is 0.56 % of the pool. Its KVCacheBlock objects take some bytes of CPU memory each; the KV data (314 MiB) is on the GPU (derived).
Deep diveReference counts and sharing, step by step
Request R1 has a 40-token prompt; R2 arrives later with the same first 32 tokens. Block size 16, prefix caching on.
| Event | R1 blocks (ref_cnt) | Free queue |
|---|---|---|
| R1 admitted: 3 new blocks a, b, c | a(1) b(1) c(1) | a, b, c removed from the front |
| R1 prefill done: a, b full, cached (c has 8 tokens) | a(1)# b(1)# c(1) | - |
| R2 admitted: hit a, b (touch), 1 new block d | a(2)# b(2)# shared; c(1); d(1) for R2 | d removed |
| R1 finishes after 4 output tokens (c not full): free c, b, a | a(1)# b(1)#; c(0) to the front | c at the front |
| R2 finishes (d not full): free d, b, a | a(0)# b(0)# to the end, d(0) to the front | d, c, ..., b, a |
# marks a block with a hash. Lines: touch (vllm/v1/core/block_pool.py:752-770), free (vllm/v1/core/block_pool.py:776-807). Block a and b now wait at the end of the queue. A third request with the same 32 tokens would hit them. A request that needs many new blocks would first take d and c, and only later evict b and a.
Deep diveErrors and edge cases
| Situation | Result | Where |
|---|---|---|
| No memory left for the KV cache at start-up | ValueError "No available memory for the cache blocks" | vllm/v1/core/kv_cache_planning.py:56-64 |
One request of max_model_len does not fit | ValueError with an estimated maximum model length | vllm/v1/core/kv_cache_planning.py:68-87 |
allocate_slots finds too few free blocks | returns None; the scheduler preempts or waits | vllm/v1/core/kv_cache_manager.py:575-579 |
get_new_blocks asked for more than are free | ValueError (the count check above prevents it) | vllm/v1/core/block_pool.py:678-679 |
reset_prefix_cache while blocks are in use | WARNING log, returns False | vllm/v1/core/block_pool.py:841-848 |
Deep diveConfiguration fields
| Flag / field | Default | Effect |
|---|---|---|
--gpu-memory-utilization | 0.92 (vllm/config/cache.py:212) | share of the GPU for this instance; the KV cache gets what is left |
--kv-cache-memory-bytes | None (vllm/config/cache.py:371) | fixed KV cache size per GPU; skips the measurement |
--num-gpu-blocks-override | None (vllm/config/cache.py:248) | replaces the computed block count (vllm/v1/core/kv_cache_planning.py:242-248) |
--block-size | 16 if no backend needs another size (see Attention backends: backend, metadata builder and implementation) | tokens per block |
--enable-prefix-caching | on for generative decoder models (see Prefix caching: the idea) | off → KVCacheCoordinatorNoPrefixCache (vllm/v1/core/kv_cache_coordinator.py:995-1008) |
watermark | 0.0 | free blocks kept when a waiting request starts (vllm/v1/core/kv_cache_manager.py:205-208) |
What can go wrong#
- Start-up fails with "No available memory for the cache blocks". The weights and the profile run used all of
gpu_memory_utilization. Raise the value, or lowermax_num_batched_tokens(it sets the size of the profile run) (vllm/v1/core/kv_cache_planning.py:56-64). - Start-up fails because one request of
max_model_lendoes not fit. Lower--max-model-lento the estimate in the error, or add memory (vllm/v1/core/kv_cache_planning.py:68-87). - Many preemptions. The pool is too small for the running requests. See The V1 scheduler: one token budget, no phases.
- Another process on the same GPU.
gpu_memory_utilizationis a share of the whole GPU, and the worker checks at start-up that this much memory is free (Executors and workers: from the engine core to the GPUs). Two instances on one GPU need two shares that add up to less than 1. - The usage metric looks low, but prefix hits drop. Cached blocks count as free (Step 8); new requests evict them from the front of the free queue.
Why it matters. The number of blocks sets how many tokens can be in flight at the same time. Too few blocks give preemptions and short batches; prefix-cache hits save blocks and prefill time. To tune memory, you must know where the block count comes from and what the usage metric counts.
Check yourself#
- 1Which process holds theeasy
KVCacheBlockobjects, and which holds the KV tensors?Show answer
The engine core process holds the
KVCacheBlockobjects in theBlockPoolof the scheduler's manager (vllm/v1/core/sched/scheduler.py:329,vllm/v1/core/block_pool.py:171-174). The workers hold the KV tensors on their GPUs and only receive block ids (vllm/v1/worker/gpu/model_runner.py:1266-1269). - 2Why does a pool of 1000 blocks give requests at most 999?easy
Show answer
The pool takes block 0 out of the free queue at start-up and makes it the null block (
vllm/v1/core/block_pool.py:184-188). The usage formula also leaves it out (vllm/v1/core/block_pool.py:888-892). - 3A running request has 32 computed tokens and 3 blocks of 16. It gets 1 new token. How many new blocks doesmedium
allocate_slotstake, and why?Show answer
None. It needs
cdiv(32 + 1, 16) = 3blocks and holds 3, so the difference is 0 (vllm/v1/core/single_type_kv_cache_manager.py:214,vllm/v1/core/single_type_kv_cache_manager.py:230-236). It needs a 4th block only when it reaches 49 tokens. - 4The usage gauge shows 0.10, but the prefix-cache hit rate falls. How can both be true?medium
Show answer
Usage counts only blocks outside the free queue (
vllm/v1/core/block_pool.py:881-892). Cached blocks withref_cnt0 are in the free queue, andget_new_blocksevicts them from the front when new requests need blocks (vllm/v1/core/block_pool.py:681-700). A low usage says nothing about how many cached prefixes survive. - 5A new request has 3 prefix-cache hit blocks that are free (ref_cnt 0) and needs 1 more block. The pool has exactly 3 free blocks. Doeshard
allocate_slotssucceed?Show answer
No. The count adds the free hit blocks to the new blocks: 1 + 3 = 4 (
vllm/v1/core/single_type_kv_cache_manager.py:256-266). The 3 free blocks in the queue are exactly the hit blocks, so4 > 3and the method returnsNone. The admission gate finds this first; the second check would give the same result (vllm/v1/core/kv_cache_manager.py:537-539,vllm/v1/core/kv_cache_manager.py:575-579). Touching the hits would take them out of the free queue, and nothing would be left for the new block.