Attention backends: backend, metadata builder and implementation

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

Attention computes softmax(QK^T)V over the cached keys and values of each request (Attention basics↗). vLLM has many attention backends: FlashAttention, FlashInfer, Triton, FlashMLA and more. Which one a layer gets is decided when the model is built (How vLLM chooses an attention backend). This page explains what a backend is made of, and how vLLM calls it each step. Kernel code is out of scope.

Each backend has three classes, with three lifetimes:

• AttentionBackend, a class with only static and class methods. It names the other two classes and answers questions: which head sizes, data types, block sizes and features it supports.
• AttentionMetadataBuilder, the metadata builder. The runner makes one per group of layers at start-up. Each step it turns the CommonAttentionMetadata into the backend's own attention metadata.
• AttentionImpl, one object per attention layer, made in the layer's constructor. Each step it writes the new keys and values into the KV cache and runs the kernel.

The split saves work. Metadata depends on the batch, not on the layer. So a 32-layer model builds it once per group and step, and all 32 layers use the same object.

An everyday comparison. Think of a train line. The timetable office (the builder) prints one timetable for all trains each day. Each driver (the impl) reads the same timetable and drives one train. The rule book (the backend class) says which trains and tracks the line can use. The comparison stops here: a builder can also say that a CUDA graph cannot replay some batches. Then vLLM changes the whole run plan at start-up.

The picture#

Read the diagram from top to bottom. Panel 1 runs once at start-up. Panel 2 runs once per KV cache group in every step. Panel 3 runs once per layer in every step, inside the forward pass.

In the code#

Step 1: the backend class names the two other classes#

Every backend implements three abstract static methods (vllm/v1/attention/backend.py:81-94). FlashAttention, for example:

    @staticmethod
    def get_name() -> str:
        return "FLASH_ATTN"

    @staticmethod
    def get_impl_cls() -> type["FlashAttentionImpl"]:
        return FlashAttentionImpl

    @staticmethod
    def get_builder_cls() -> type["FlashAttentionMetadataBuilder"]:
        return FlashAttentionMetadataBuilder

The rest of the class answers questions. The supports_* class methods and validate_configuration are used during selection (How vLLM chooses an attention backend, vllm/v1/attention/backend.py:279-370). Four more members matter at run time:

Step 2: one impl per layer#

The Attention constructor makes the impl right after the backend is selected (vllm/model_executor/layers/attention/attention.py:419-432). The impl gets the per-rank head counts, the head size, the scale, the sliding window and the KV cache data type. The abstract forward has this signature:

    @abstractmethod
    def forward(
        self,
        layer: AttentionLayer,
        query: torch.Tensor,
        key: torch.Tensor,
        value: torch.Tensor,
        kv_cache: torch.Tensor,
        attn_metadata: T,
        output: torch.Tensor,
        output_scale: torch.Tensor | None = None,
        output_block_scale: torch.Tensor | None = None,
    ) -> torch.Tensor:
        raise NotImplementedError

query is [num_tokens, num_heads, head_size]; the layer reshapes q, k and v before the op (vllm/model_executor/layers/attention/attention.py:533-540). The impl writes into output; it does not return a new tensor. Some backends set forward_includes_kv_cache_update = False. They also implement do_kv_cache_update(layer, key, value, kv_cache, slot_mapping), which writes the keys and values of the new tokens into their slots. For FlashAttention it is a scatter by slot (vllm/v1/attention/backends/flash_attn.py:1430-1464).

In the profile run there is no metadata. FlashAttention then only fills the output with zeros (vllm/v1/attention/backends/flash_attn.py:1224-1226).

Step 3: attention groups and builders, at start-up#

When the KV cache is initialized, init_attn_backend groups the layers. Inside each KV cache group, layers with the same backend class, the same KV cache spec and the same number of query heads form one AttentionGroup:

            layer_kv_cache_spec: KVCacheSpec = kv_cache_group_spec.kv_cache_spec
            if isinstance(layer_kv_cache_spec, UniformTypeKVCacheSpecs):
                layer_kv_cache_spec = layer_kv_cache_spec.kv_cache_specs[layer_name]

            # Split on per-rank num_heads_q so layers with different Q-head
            # counts (e.g. a spec-decode draft head and its target) get separate
            # metadata builders.
            num_heads_q = getattr(attn_layers[layer_name], "num_heads", 0)
            key = (attn_backend.full_cls_name(), layer_kv_cache_spec, num_heads_q)
            if key not in group_map:
                group_map[key] = AttentionGroup(
                    attn_backend, [layer_name], layer_kv_cache_spec, kv_cache_group_id
                )
                group_order.append(key)
            else:
                group_map[key].layer_names.append(layer_name)

        attn_groups.append([group_map[key] for key in group_order])

Then each group makes its builders. It passes the KV cache spec (with the kernel block size), the layer names, the config and the device (vllm/v1/worker/gpu/attn_utils.py:291-300, vllm/v1/worker/utils.py:330-339). Normally there is one builder per group; with dual-batch overlap there is one per micro-batch, because "some builders keep the prepared metadata on themselves" (vllm/v1/worker/gpu/attn_utils.py:296-299).

Step 4: CommonAttentionMetadata, once per KV cache group and step#

build_attn_metadata loops over the KV cache groups. For each group it makes one CommonAttentionMetadata from the step's input tensors (The model runner (Model Runner V2)):

        common_attn_metadata = CommonAttentionMetadata(
            query_start_loc=query_start_loc_gpu,
            query_start_loc_cpu=query_start_loc_cpu,
            seq_lens=seq_lens,
            seq_lens_cpu_upper_bound=seq_lens_cpu_upper_bound,
            max_seq_len=max_seq_len,
            num_reqs=num_reqs,
            num_actual_tokens=num_tokens,
            max_query_len=max_query_len,
            block_table_tensor=block_table,
            slot_mapping=slot_mapping,
            causal=group_causal,
            dcp_local_seq_lens=dcp_local_seq_lens,
            dcp_local_seq_lens_cpu_upper_bound=dcp_local_seq_lens_cpu_upper_bound,
            positions=positions,
            is_prefilling=group_is_prefilling,
            mm_req_doc_ranges=mm_req_doc_ranges,
            rswa_prefix_lens=rswa_prefix_lens,
            req_idx=req_idx,
            _token_to_req_indices_cache=token_to_req_indices,
            **common_attn_metadata_extra_kwargs,
        )

Mainly the block table and the slot mapping differ between groups: each group has its own block table and its own row of slot mappings (vllm/v1/worker/gpu/attn_utils.py:504-505). A hybrid drafter can also set a causal flag per group, and a model state can add its own fields (vllm/v1/worker/gpu/attn_utils.py:506-520). The main fields:

Definitions: vllm/v1/attention/backend.py:389-475.

Step 5: builder.build, and one metadata object for many layers#

For each attention group of the KV cache group, the runner calls the builder and stores the result under every layer name of the group:

        for attn_group in attn_groups[i]:
            attn_metadata_builder = attn_group.get_metadata_builder(ubatch_idx)
            reuse_key = None
            if attn_metadata_builder.supports_update_block_table and isinstance(
                attn_group.kv_cache_spec, MambaSpec
            ):
                reuse_key = (attn_group.kv_cache_spec, type(attn_metadata_builder))
            if reuse_key in cached_metadata:
                metadata = attn_metadata_builder.update_block_table(
                    cached_metadata[reuse_key], block_table, slot_mapping
                )
            elif for_cudagraph_capture:
                metadata = attn_group.build_metadata_for_cudagraph_capture(
                    common_attn_metadata, ubatch_idx
                )
            else:
                attn_metadata_extra_kwargs = (
                    model_specific_attn_metadata.get_extra_attn_kwargs(
                        attn_metadata_builder,
                        num_reqs,
                    )
                    if model_specific_attn_metadata is not None
                    else {}
                )
                metadata = attn_group.build_metadata(
                    common_attn_metadata, ubatch_idx, **attn_metadata_extra_kwargs
                )
            if reuse_key is not None and reuse_key not in cached_metadata:
                cached_metadata[reuse_key] = metadata
            for layer_name in attn_group.layer_names:
                attn_metadata[layer_name] = metadata

1. Lines 553-560. Mamba groups with the same spec and builder reuse the first metadata and only swap the block table, through update_block_table.
2. Lines 561-564. During CUDA graph capture, build_for_cudagraph_capture is called instead. Its default just calls build (vllm/v1/attention/backend.py:720-729).
3. Lines 574-576. The normal path: AttentionGroup.build_metadata calls build(common_prefix_len=0, common_attn_metadata) (vllm/v1/worker/utils.py:341-352). The common_prefix_len argument is "Deprecated and always 0" (vllm/v1/attention/backend.py:686-704).
4. Lines 579-580. The same object goes under every layer name of the group.

If the backend's kernel uses smaller blocks than the KV cache manager, the attention group first rewrites the block table into kernel blocks (vllm/v1/worker/utils.py:368-378).

Step 6: CUDA graph support, and how it lowers the mode#

A builder reports how much CUDA graph capture it allows, with AttentionCGSupport:

class AttentionCGSupport(Enum):
    """Constants for the cudagraph support of the attention backend"""

    ALWAYS = 3
    """Cudagraph always supported; supports mixed-prefill-decode"""
    UNIFORM_BATCH = 2
    """Cudagraph supported for batches the only contain query lengths that are
    the same, this can be used for spec-decode
        i.e. "decodes" are 1 + num_speculative_tokens"""
    UNIFORM_SINGLE_TOKEN_DECODE = 1
    """Cudagraph supported for batches the only contain query_len==1 decodes"""
    NEVER = 0
    """NO cudagraph support"""

The default is NEVER (vllm/v1/attention/backend.py:593-595). The runner asks every group and keeps the weakest answer (vllm/v1/worker/gpu/attn_utils.py:314-341). Then resolve_cudagraph_mode_and_sizes may lower the configured mode (vllm/v1/worker/gpu/model_runner.py:754-763):

1. FULL for mixed batches, but support below ALWAYS. NEVER raises ValueError. Otherwise the mode becomes FULL_AND_PIECEWISE if attention is a splitting op, else FULL_DECODE_ONLY (vllm/config/compilation.py:1394-1418).
2. FULL decode graphs, but support NEVER. The mode becomes PIECEWISE if attention is compiled piecewise, else NONE (vllm/config/compilation.py:1421-1445).
3. FULL decode graphs with speculative decoding (decode query length above 1), but support below UNIFORM_BATCH. The mode becomes PIECEWISE or NONE in the same way (vllm/config/compilation.py:1448-1465).
4. Last check: a mode with FULL graphs and support NEVER raises (vllm/config/compilation.py:1482-1493).

Each change logs a warning that names the backend. torch.compile and CUDA graphs in vLLM explains the modes.

Step 7: the MLA variant#

Models with multi-head latent attention, such as DeepSeek-V3, use the layer MLAAttention and an impl based on MLAAttentionImpl. Its constructor gets the MLA dimensions and the kv_b_proj layer (vllm/v1/attention/backend.py:1034-1064). The impl has two paths instead of one forward:

• forward_mqa for decode tokens, over the compressed latent cache (vllm/v1/attention/backend.py:1080-1089);
• forward_mha for prefill tokens, "MHA-style" (vllm/v1/attention/backend.py:1066-1079).

The layer sends the first num_decode_tokens rows to forward_mqa and the rest to forward_mha (vllm/model_executor/layers/attention/mla_attention.py:984-985, vllm/model_executor/layers/attention/mla_attention.py:1017-1020). The MLA builder finds the split with split_decodes_and_prefills (vllm/model_executor/layers/attention/mla_attention.py:2605-2612). This is why Model Runner V2 puts the decodes first in the batch (vllm/v1/worker/gpu/model_runner.py:2476-2483). The cache write is one shared method, do_kv_cache_update, which calls the concat_and_cache_mla op (vllm/v1/attention/backend.py:1108-1128). The custom ops are unified_mla_kv_cache_update and unified_mla_attention_with_output (vllm/model_executor/layers/attention/mla_attention.py:1475-1479, vllm/model_executor/layers/attention/mla_attention.py:1524-1530).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

class FlashAttentionMetadata:
    # NOTE(sang): Definition of context_len, query_len, and seq_len.
    # |---------- N-1 iteration --------|
    # |---------------- N iteration ---------------------|
    # |- tokenA -|......................|-- newTokens ---|
    # |---------- context_len ----------|
    # |-------------------- seq_len ---------------------|
    #                                   |-- query_len ---|

    num_actual_tokens: int  # Number of tokens excluding padding.
    max_query_len: int
    query_start_loc: torch.Tensor
    max_seq_len: int
    seq_lens: torch.Tensor
    block_table: torch.Tensor
    slot_mapping: torch.Tensor

    # For GQA DCP
    max_dcp_context_kv_len: int | None = None
    dcp_context_kv_lens: torch.Tensor | None = None

    # Split counts for FA2 DCP context attention. num_prefill_* tracks
    # context-bearing extend rows; pure prefills do not attend to DCP context.
    num_decode_reqs: int = 0
    num_prefill_reqs: int = 0
    num_decode_tokens: int = 0
    num_prefill_tokens: int = 0

    # Optional aot scheduling
    scheduler_metadata: torch.Tensor | None = None
    max_num_splits: int = 0

    causal: bool | torch.Tensor = True

What can go wrong#

• A backend with no CUDA graph support and -cc.cudagraph_mode=FULL. Start-up stops with "CUDAGraphMode.FULL is not supported with ... backend (support: NEVER); please try cudagraph_mode=PIECEWISE" (vllm/config/compilation.py:1400-1410).
• Silent mode changes. A weaker backend lowers FULL_AND_PIECEWISE to PIECEWISE or NONE with only a warning. Decode then loses its full graph and gets slower. Look for "is not supported with" in the log.
• Batch invariance. With VLLM_BATCH_INVARIANT=1, a backend without supports_batch_invariance() raises RuntimeError in init_attn_backend (vllm/v1/worker/gpu/attn_utils.py:235-242).
• A layout that some backend cannot read. VLLM_KV_CACHE_LAYOUT outside the supported set raises ValueError; backends with no common layout raise too (vllm/v1/attention/backends/utils.py:361-367, vllm/v1/attention/backends/utils.py:271-276).
• Fused output quantization on FlashAttention. Its forward raises NotImplementedError when it gets an output scale (vllm/v1/attention/backends/flash_attn.py:1219-1222).

Why it matters. A backend plugin implements these three classes. The builder decides the CPU cost of each step and whether decode can use a FULL CUDA graph; the impl decides the GPU cost. When a deployment is slower than expected, check which backend each layer got and which CUDA graph mode survived start-up.