KV connectors

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

The KV cache keeps the keys and values of earlier tokens, so decode does not compute them again (The KV cache↗). Prefix caching reuses these blocks inside one vLLM instance (Prefix caching in the engine core: hashes, lookup, caching, eviction and KV events). A KV connector goes further: it moves blocks to or from something outside the instance. Examples are a second vLLM instance (disaggregated prefill), CPU memory, a local disk or a distributed store.

The work has two parts that run in different processes. The scheduler must know, before it allocates blocks, how many prompt tokens it can load instead of compute. The workers must copy the bytes into or out of the GPU. So one connector class is made twice, with a different connector role. The SCHEDULER object lives in the EngineCore process. The WORKER object lives in each worker. Each step, the scheduler side writes a metadata object; the worker side reads it.

You turn a connector on with --kv-transfer-config, a KVTransferConfig. Its field kv_connector names the class. KVConnectorFactory maps names to classes. vLLM registers 17 built-in connectors, and a plugin can register more, or name a module path.

An everyday comparison. Think of a restaurant kitchen with an external store room. Before each round of orders, the head chef checks the store list for the sauces of each order. If they are in the store, she reserves a shelf and writes a pick list. The kitchen porters read the list, bring the items to the shelf, and take finished items back to the store. The comparison stops here. In vLLM the porters (workers) can also carry items layer by layer during cooking, and the chef can postpone an order until its items arrive.

The picture#

Follow the numbers. Steps 1-3 run on the CPU in the engine core, before the step goes to the workers. Steps 4-6 run in every worker around the forward pass. Steps 7-9 run in the engine core when the outputs come back, and in the next schedule.

In the code#

Step 1: the configuration#

@config
class KVTransferConfig:
    """Configuration for distributed KV cache transfer."""

    kv_connector: str | None = None
    """The KV connector for vLLM to transmit KV caches between vLLM instances.
    """

    engine_id: str | None = None
    """The engine id for KV transfers."""

    kv_buffer_device: str = field(default_factory=kv_buffer_device_default_factory)
    """The device used by kv connector to buffer the KV cache. Choices are
    'cuda', 'cpu' and 'xpu'."""

    kv_role: KVRole | None = None
    """Whether this vLLM instance produces, consumes KV cache, or both. Choices
    are 'kv_producer', 'kv_consumer', and 'kv_both'."""

    kv_port: int = 14579
    """The KV connector port, used to build distributed connection."""

    kv_connector_extra_config: dict[str, Any] = field(default_factory=dict)
    """any extra config that the connector may need."""

    kv_connector_module_path: str | None = None
    """The Python module path to dynamically load the KV connector from.
    Only supported in V1."""

    enable_permute_local_kv: bool = False
    """Experiment feature flag to enable HND to NHD KV Transfer"""

    kv_load_failure_policy: Literal["recompute", "fail"] = "fail"
    """Policy for handling KV cache load failures.
    'recompute': reschedule the request to recompute failed blocks
    'fail': immediately fail the request with an error finish reason (default)"""

• kv_connector is the registered name or the class name. kv_role is kv_producer, kv_consumer or kv_both; a connector without a role raises (vllm/config/kv_transfer.py:120-124).
• kv_connector_extra_config is a free dict for the connector, for example cpu_bytes_to_use.
• kv_load_failure_policy defaults to "fail": a failed load fails the request.
• engine_id becomes a random UUID when unset (vllm/config/kv_transfer.py:111-112).

The producer and consumer properties decide some scheduler behaviour (vllm/config/kv_transfer.py:126-136). A typical disaggregated setup runs one instance with kv_producer and one with kv_consumer; the page Disaggregated prefill: prefill and decode on different instances explains it. This command is adapted from the docs (docs/features/disagg_prefill.md:41):

vllm serve Qwen/Qwen3-8B --kv-transfer-config \
  '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":1000000000}}'

Step 2: the factory finds the class#

    @classmethod
    def get_connector_class(
        cls, kv_transfer_config: "KVTransferConfig"
    ) -> type[KVConnectorBaseType]:
        connector_name = kv_transfer_config.kv_connector
        if connector_name is None:
            raise ValueError("Connector name is not set in KVTransferConfig")
        connector_module_path = kv_transfer_config.kv_connector_module_path
        if connector_module_path is not None and not connector_module_path:
            raise ValueError("kv_connector_module_path cannot be an empty string.")
        if connector_module_path:
            # External module path takes priority over internal registry.
            connector_module = importlib.import_module(connector_module_path)
            try:
                connector_cls = getattr(connector_module, connector_name)
            except AttributeError as e:
                raise AttributeError(
                    f"Class {connector_name} not found in {connector_module_path}"
                ) from e
            connector_cls = cast(type[KVConnectorBaseType], connector_cls)
            if not supports_kw(connector_cls, "kv_cache_config"):
                msg = (
                    f"Connector {connector_cls.__name__} uses deprecated "
                    "2-argument constructor signature. External v1 KV "
                    "connectors must accept kv_cache_config as the third "
                    "constructor argument and pass it to super().__init__()."
                )
                logger.error(msg)
                raise ValueError(msg)
        elif connector_name in cls._registry:
            connector_cls = cls._registry[connector_name]()
        else:
            raise ValueError(f"Unsupported connector type: {connector_name}")
        return connector_cls

1. Lines 106-124. When kv_connector_module_path is set, it wins over the registry. vLLM imports the module and takes the class named kv_connector. A class without the kv_cache_config constructor argument raises.
2. Lines 125-126. Else the registry: it calls the stored loader, which imports the module only now.
3. Lines 127-128. An unknown name raises "Unsupported connector type".

A plugin registers a name with register_connector(name, module_path, class_name). A name that exists raises ValueError (vllm/distributed/kv_transfer/kv_connector/factory.py:30-40). The built-in names are registered when the factory module is imported (lines 149-249). The page distributed/kv_transfer/kv_connector/factory.py: the KV connector registry reads the file line by line.

Step 3: one class, two objects#

The scheduler makes the SCHEDULER object in its constructor:

        kv_transfer_config = self.vllm_config.kv_transfer_config
        if kv_transfer_config is not None:
            assert not self.is_encoder_decoder, (
                "Encoder-decoder models are not currently supported with KV connectors"
            )
            self.connector = KVConnectorFactory.create_connector(
                config=self.vllm_config,
                role=KVConnectorRole.SCHEDULER,
                kv_cache_config=self.kv_cache_config,
            )
            if self.log_stats:
                self.connector_prefix_cache_stats = PrefixCacheStats()
            kv_load_failure_policy = kv_transfer_config.kv_load_failure_policy
            self.recompute_kv_load_failures = kv_load_failure_policy == "recompute"

            # With overlapping batches (async scheduling or PP), a step may
            # still be writing a freed request's KV blocks. A consumer KV
            # Connector can reallocate and fill those blocks via a load that
            # isn't ordered against that write, so defer freeing them.
            multiple_inflight_batches = self.vllm_config.max_concurrent_batches > 1
            if multiple_inflight_batches and kv_transfer_config.is_kv_consumer:
                self.defer_block_free = True

            self.requires_kv_delivery = self.connector.requires_kv_delivery

Each worker makes the WORKER object after it knows the KV cache config (vllm/v1/worker/gpu_worker.py:873-878, vllm/distributed/kv_transfer/kv_transfer_state.py:73-92). The roles are an enum with two values (vllm/distributed/kv_transfer/kv_connector/v1/base.py:139-144). The factory checks the hybrid KV cache manager: a connector that does not support it raises when the manager is on (vllm/distributed/kv_transfer/kv_connector/factory.py:54-60). VllmConfig turns the manager off for such a connector when the user did not choose, with a warning (vllm/config/vllm.py:2236-2259).

The base class constructor stores the config and logs a warning that the API "is experimental and subject to change" (vllm/distributed/kv_transfer/kv_connector/v1/base.py:214-232).

Step 4: the scheduler-side methods#

Three methods are abstract on the scheduler side (vllm/distributed/kv_transfer/kv_connector/v1/base.py:515-611):

Other scheduler-side methods have defaults: request_finished (no delay), update_connector_output, on_new_request, take_events and more (vllm/distributed/kv_transfer/kv_connector/v1/base.py:613-680).

Step 5: the worker-side methods#

Four methods are abstract on the worker side (vllm/distributed/kv_transfer/kv_connector/v1/base.py:332-396): start_load_kv, wait_for_layer_load, save_kv_layer and wait_for_save. Model Runner V2 wraps them in a small class:

    def pre_forward(
        self,
        scheduler_output: "SchedulerOutput",
        input_batch: "InputBatch | None" = None,
        attn_metadata: dict[str, Any] | None = None,
    ) -> None:
        if self._disabled:
            return

        kv_connector_metadata = scheduler_output.kv_connector_metadata
        assert kv_connector_metadata is not None
        self.kv_connector.handle_preemptions(kv_connector_metadata)
        self.kv_connector.bind_connector_metadata(kv_connector_metadata)
        self._pending_load_kwargs = self._build_load_kwargs(
            scheduler_output, input_batch, attn_metadata
        )

        if scheduler_output.has_sync_kv_loads:
            # Sync loads need to run before this step's forward.
            self._start_load_kv()
        # Otherwise start the async load after forward to keep submission
        # off the critical path.

A synchronous load must finish before the forward reads the blocks, so it starts in pre_forward. An asynchronous load starts after the forward, "to keep submission off the critical path". The layer-by-layer calls wrap the attention custom op:

    @wraps(func)
    def wrapper(*args, **kwargs):
        if not has_kv_transfer_group() or not is_v1_kv_transfer_group():
            return func(*args, **kwargs)

        layer_name = _resolve_layer_name(args[layer_name_index])

        # Extract attention context (metadata, layer, kv_cache, layer_slot_mapping)
        attn_metadata, _, kv_cache, _ = get_attention_context(layer_name)
        connector = get_kv_transfer_group()
        if attn_metadata is None or not connector.has_connector_metadata():
            return func(*args, **kwargs)

        # Wait for KV layer on entry
        connector.wait_for_layer_load(layer_name)

        # Execute the function
        result = func(*args, **kwargs)

        # Save KV cache layer on exit
        connector.save_kv_layer(layer_name, kv_cache, attn_metadata)

        return result

    return wrapper

After the forward, post_forward calls wait_for_save and collects the finished transfers into a KVConnectorOutput (vllm/v1/worker/gpu/kv_connector.py:136-160). With speculative decoding, the save finishes later, after the drafter runs (vllm/v1/worker/gpu/model_runner.py:2220-2224). The model runner calls it on every pipeline rank (vllm/v1/worker/gpu/model_runner.py:2150, vllm/v1/worker/gpu/model_runner.py:2222-2224).

Step 6: async loads, delayed caching and deferred frees#

The scheduler integrates the connector in four places:

1. External prefix hits. The connector sees a block-aligned local hit and returns more tokens. The scheduler counts them as computed (vllm/v1/core/sched/scheduler.py:1027-1039).
2. Async loads. For an async load, allocate_slots gets delay_cache_blocks=True (vllm/v1/core/sched/scheduler.py:1327). The block hashes are not committed yet, because the bytes are not there (vllm/v1/core/kv_cache_manager.py:602-605). The request goes to WAITING_FOR_REMOTE_KVS and is skipped for this step (vllm/v1/core/sched/scheduler.py:1384-1411).
3. Completion. When a worker reports finished_recving, the request is marked ready (vllm/v1/core/sched/scheduler.py:3325-3334). The next schedule caches the loaded blocks and moves it back to WAITING (vllm/v1/core/sched/scheduler.py:3275-3289, vllm/v1/core/sched/scheduler.py:3254-3257).
4. Frees. request_finished can return True (a connector with SupportsHMA gets request_finished_all_groups, vllm/v1/core/sched/scheduler.py:3145-3157): the blocks stay until the worker reports finished_sending (vllm/v1/core/sched/scheduler.py:2840-2842, vllm/v1/core/sched/scheduler.py:3335-3338). Separately, defer_block_free holds freed blocks while an in-flight step may still write them. It is on only with more than one batch in flight and a consumer role (vllm/v1/core/sched/scheduler.py:187-193).

Step 7: which processes read the registry#

The registry is a class attribute in each process. Three processes read it, for different reasons:

So a connector registered by a plugin must be registered in all three processes. A general plugin does that. kv_connector_module_path needs no registration, because each process imports the module itself.

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

        # KV offloading is only activated when kv_offloading_size is set.
        if (kv_offloading_size := self.cache_config.kv_offloading_size) is None:
            return

        kv_offloading_backend = self.cache_config.kv_offloading_backend

        # If no KVTransferConfig is provided, create a default one.
        if self.kv_transfer_config is None:
            self.kv_transfer_config = KVTransferConfig()

        if kv_offloading_backend == "native":
            if envs.VLLM_USE_SIMPLE_KV_OFFLOAD:
                config_connector = "SimpleCPUOffloadConnector"
            else:
                config_connector = "OffloadingConnector"
            self.kv_transfer_config.kv_connector = config_connector
            self.kv_transfer_config.kv_connector_extra_config.update(
                {"cpu_bytes_to_use": kv_offloading_size * (1 << 30)}
            )
        elif kv_offloading_backend == "lmcache":
            # Default to LMCache multi-process (MP) mode. The actual KV
            # storage capacity is managed by the standalone LMCache server
            # process, so ``kv_offloading_size`` is not propagated here.
            # ``LMCacheMPConnector`` falls back to ``tcp://localhost:5555``
            # when host/port are not provided via extra_config.
            self.kv_transfer_config.kv_connector = "LMCacheMPConnector"

        # This is the same for all backends
        self.kv_transfer_config.kv_role = "kv_both"

What can go wrong#

• The front end does not know your connector. The front end reads the class for config checks (vllm/config/vllm.py:2243). If only the engine core and the workers register it, VllmConfig or the metrics loggers fail with "Unsupported connector type". Use a general plugin or kv_connector_module_path.
• Old constructor. A module-path connector must accept kv_cache_config as the third argument, else ValueError (vllm/distributed/kv_transfer/kv_connector/factory.py:116-124).
• Hybrid models. A connector without SupportsHMA turns the hybrid KV cache manager off, and the warning says that hybrid SSM models "will fail at startup without it" (vllm/config/vllm.py:2245-2256).
• Encoder-decoder models. The scheduler asserts "Encoder-decoder models are not currently supported with KV connectors" (vllm/v1/core/sched/scheduler.py:174-176).
• Backends that refuse a connector. An attention backend can reject a KV connector in its validation; ROCM_ATTN does (vllm/v1/attention/backends/rocm_attn.py:184-188). See How vLLM chooses an attention backend.
• Load failures. With the default policy "fail", a failed load fails the request; "recompute" computes the missing blocks instead (vllm/config/kv_transfer.py:88-91, vllm/v1/core/sched/scheduler.py:184-185).

Why it matters. KV connectors are the base of disaggregated prefill and of KV offloading, two of the main ways to scale online serving. Most connector problems are process problems: which process registered the class, and which role runs where.