Serving LoRA adapters: from the model name to the wrapped layer

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

LoRA adapters explains what a LoRA adapter is, why vLLM keeps adapters unmerged, and the flags with their real defaults (--enable-lora off, --max-loras 1, --max-lora-rank 16). This page follows one adapter through the processes of vllm serve.

Four parts work together. The API server turns the model field into a LoRARequest. The scheduler admits a request only if the step does not exceed max_loras adapters. The worker keeps adapters in two LRU caches: one in CPU memory, and a fixed number of GPU slots. At model load, vLLM replaces each supported layer with a LoRA wrapper. The wrapper runs the base layer once for the batch and then calls the Punica wrapper kernels, which add each token's own adapter update.

An everyday comparison. Think of a print shop with one press and a rack of three stencil holders. The front desk writes the customer's stencil name on each order. The planner never puts more than three stencils into one run. The stock room keeps stencils on shelves and moves one into a holder when a run needs it. The comparison stops being true in one place: the shop moves a stencil into a holder only when a run needs it. vLLM also activates an adapter at once when you load it through the API.

The picture#

Read the panels from top to bottom. Panel 1 runs in three processes: the API server, the engine core and the worker. Panels 2 and 3 run in the worker in every step. The slot layout [1, 3, 2] is one possible order, for example after an LRU eviction; it depends on when each adapter was activated. The worked example in the deep dive gets [1, 2, 3] from its start-up order.

In the code#

Step 1: the request gets a LoRARequest#

The model field of a request selects the base model or an adapter; _maybe_get_adapters looks the name up (vllm/entrypoints/serve/engine/serving.py:194-200). The result is a small msgspec struct:

class LoRARequest(
    msgspec.Struct,
    omit_defaults=True,  # type: ignore[call-arg]
    array_like=True,
):  # type: ignore[call-arg]
    """Request for a LoRA adapter.

    lora_int_id must be globally unique for a given adapter.
    This is currently not enforced in vLLM.

    load_inplace: If True, forces reloading the adapter even if one
        with the same lora_int_id already exists in the cache. This replaces
        the existing adapter in-place. If False (default), only loads if the
        adapter is not already loaded.
    """

    lora_name: str
    lora_int_id: int
    lora_path: str = ""
    base_model_name: str | None = msgspec.field(default=None)
    load_inplace: bool = False
    is_3d_lora_weight: bool = False

lora_int_id must be above 0 (vllm/lora/request.py:37-39). Two LoRA requests are equal when their names are equal; the hash also uses only the name (vllm/lora/request.py:56-69). An unknown name gives HTTP 404. Only with VLLM_ALLOW_RUNTIME_LORA_UPDATING does the server first ask the LoRA resolvers (vllm/entrypoints/serve/engine/serving.py:62-66; Plugins in the API server, Step 4).

Step 2: the scheduler limits adapters per step#

At the start of the waiting pass, the scheduler collects the adapter ids of all running requests. This includes requests that are not scheduled in this step (vllm/v1/core/sched/scheduler.py:945-954). A waiting request with a new adapter is skipped when that set already has max_loras entries:

                # Check that adding the request still respects the max_loras constraint.
                if (
                    self.lora_config
                    and request.lora_request
                    and (
                        len(scheduled_loras) == self.lora_config.max_loras
                        and request.lora_request.lora_int_id not in scheduled_loras
                    )
                ):
                    # Scheduling would exceed max_loras, skip.
                    skip_request(request_queue)

The skipped request goes back to the front of the queue and tries again in the next step. The gauge vllm:lora_requests_info shows max_lora and the waiting and running adapters as labels (vllm/v1/metrics/loggers.py:1099-1112).

Step 3: the worker loads and activates adapters#

The LoRA manager exists only with --enable-lora. load_lora_model checks that the model supports LoRA, builds an LRUCacheWorkerLoRAManager, and wraps the model (vllm/v1/worker/lora_model_runner_mixin.py:43-66). In each step with LoRA requests, the model runner passes the active LoRARequests to the manager (vllm/v1/worker/lora_model_runner_mixin.py:68-87):

    def _apply_adapters(self, lora_requests: set[LoRARequest]) -> None:
        loras_map = {
            lora_request.lora_int_id: lora_request
            for lora_request in lora_requests
            if lora_request
        }
        if len(loras_map) > self._adapter_manager.lora_slots:
            raise RuntimeError(
                f"Number of requested LoRAs ({len(loras_map)}) is greater "
                "than the number of GPU LoRA slots "
                f"({self._adapter_manager.lora_slots})."
            )
        for lora in loras_map.values():
            self.add_adapter(lora)

    def add_adapter(self, lora_request: LoRARequest) -> bool:
        # Note that this method is not thread-safe. It may be invoked multiple
        # times for the same adapter when using multiple API servers.
        # This is ok because it's currently only called from
        # the single-threaded core engine loop.

        with gpu_sync_allowed():
            if (
                lora_request.lora_int_id not in self.list_adapters()
                or lora_request.load_inplace
            ):
                # Load the new adapter first to ensure it is actually valid, before
                # evicting any existing adapters.
                # This may cause the # of loaded lora adapters to very temporarily
                # exceed `--max-cpu-loras`.
                lora = self._load_adapter(lora_request)

                # Remove the existing adapter if it exists
                # Use case for LoRA inplace
                self._adapter_manager.remove_adapter(lora.id)

                # Loading succeeded, now check if we will exceed cache capacity and
                # evict if the oldest adapter if so
                if len(self._adapter_manager) + 1 > self._adapter_manager.capacity:
                    assert isinstance(self._adapter_manager, LRUCacheLoRAModelManager)
                    self._adapter_manager.remove_oldest_adapter()
                # Then add the new adapter to the cache
                loaded = self._adapter_manager.add_adapter(lora)
            else:
                # If the lora is already loaded, just touch it to
                # update its position in the caches
                loaded = (
                    self._adapter_manager.get_adapter(lora_request.lora_int_id)
                    is not None
                )
            self._adapter_manager.activate_adapter(lora_request.lora_int_id)
        return loaded

1. Lines 279-284: more adapters than GPU slots in one batch raises RuntimeError. The scheduler check of Step 2 prevents this.
2. Lines 295-303: an adapter that is not in the CPU cache is loaded from disk first.
3. Lines 309-313: when the CPU cache is full (max_cpu_loras), the oldest adapter is removed.
4. Line 323: activate_adapter puts the adapter into a GPU slot.

Loading. _load_adapter resolves the path. An absolute path is used as is. A path that does not exist locally is a Hugging Face Hub repository, and vLLM downloads it. With VLLM_USE_MODELSCOPE, it is a ModelScope repository (vllm/lora/utils.py:358-385). Then it checks the rank against max_lora_rank and loads the weights to CPU memory in the LoRA dtype (vllm/lora/worker_manager.py:119-150). When the adapter enters the cache, alpha / r is merged into B, so the kernels use scale 1.0 (vllm/lora/model_manager.py:907-908, vllm/lora/lora_weights.py:37-42).

Activation. activate_adapter takes the first free slot and copies each module's A and B into the stacked buffers at that index (vllm/lora/model_manager.py:355-412). The LRU manager first deactivates the oldest active adapter when all slots are used (vllm/lora/model_manager.py:1440-1452). A module without weights in the adapter gets its slot zeroed (reset_lora).

Step 4: the wrapped layers#

At model load, the LoRA model manager walks over all modules. Without --lora-target-modules, it wraps every supported module (vllm/lora/model_manager.py:476-500). from_layer picks the first LoRA class whose can_replace_layer accepts the layer, and allocates its buffers for max_loras slots (vllm/lora/utils.py:113-131). A BaseLayerWithLoRA keeps the original layer as base_layer and forwards unknown attributes to it (vllm/lora/layers/base.py:17-34). The linear wrappers run the base layer, then add the LoRA output:

    def _apply_sync(
        self, x: torch.Tensor, bias: torch.Tensor | None = None
    ) -> torch.Tensor:
        output = self._get_quant_method().apply(self.base_layer, x, bias)
        return self._apply_lora_to_output(x, output)

    def _apply_base_forward(self, x: torch.Tensor) -> torch.Tensor:
        base_output = self.base_layer(x)
        output = base_output[0] if isinstance(base_output, tuple) else base_output
        return self._apply_lora_to_output(x, output)

    def _apply_lora_to_output(
        self, x: torch.Tensor, output: torch.Tensor
    ) -> torch.Tensor:
        original_shape = output.shape if output.ndim == 3 else None

        # In transformers backend, x and output have extra batch dimension like
        # (1, seq_len, hidden_dim), while punica expects (seq_len, hidden_dim),
        # therefore we need to flatten the batch dimensions.
        if x.ndim == 3 and output.ndim == 3:
            output = output.flatten(0, 1)
            x = x.flatten(0, 1)

        lora_output: torch.Tensor | None = self.punica_wrapper.add_lora_linear(
            output, x, self.lora_a_stacked, self.lora_b_stacked, 1.0, self.output_slices
        )

The Punica wrapper wrapper class comes from the platform: PunicaWrapperGPU on CUDA (vllm/platforms/cuda.py:577-578, vllm/lora/punica_wrapper/punica_selector.py:13-21). add_lora_linear first runs a "shrink" (x @ A, into an FP32 buffer of rank width) and then an "expand" (buffer @ B, added to the output) (vllm/lora/punica_wrapper/punica_gpu.py:218-262). The kernels themselves are Triton code and out of scope.

Step 5: token_lora_mapping in Model Runner V2#

LoraState keeps one LoRA id per request slot; 0 means no adapter (vllm/v1/worker/gpu/lora_utils.py:75-89). In execute_model, it builds the per-token map and activates the adapters:

    def make_lora_inputs(
        self,
        req_ids: list[str],
        idx_mapping: np.ndarray,
        num_scheduled_tokens: np.ndarray,
    ) -> tuple[tuple[int, ...], tuple[int, ...], set[LoRARequest]]:
        lora_ids = self.lora_ids[idx_mapping]
        prompt_lora_mapping = tuple(lora_ids)
        token_lora_mapping = tuple(lora_ids.repeat(num_scheduled_tokens))
        active_lora_requests: set[LoRARequest] = self.get_activate_loras(req_ids)
        return prompt_lora_mapping, token_lora_mapping, active_lora_requests

            if self.lora_config:
                # Activate LoRA adapters.
                lora_inputs = self.lora_state.make_lora_inputs(
                    input_batch.req_ids,
                    input_batch.idx_mapping_np,
                    input_batch.num_scheduled_tokens,
                )
                self._set_active_loras(*lora_inputs)

Punica turns LoRA ids into slot indices; a token without an adapter gets -1 (vllm/lora/punica_wrapper/utils.py:87-111). set_adapter_mapping rebuilds the metadata only when the mapping or the slot layout changed since the last step (vllm/lora/model_manager.py:1401-1409).

Step 6: CUDA graphs with LoRA#

A CUDA graph fixes the number of active adapters it was captured with. So Model Runner V2 captures every capture size once per "LoRA case" (vllm/v1/worker/gpu/cudagraph_utils.py:305-307), and a graph matches a batch only with the same case (vllm/v1/worker/gpu/cudagraph_utils.py:136):

def get_lora_capture_cases(
    lora_config: "LoRAConfig | None",
    compilation_config: "CompilationConfig",
) -> list[int]:
    """Return num_active_loras values for cudagraph capture.

    When cudagraph_specialize_lora=True: powers of 2 up to max_loras, plus
    max_loras+1. When False: [0, max_loras+1]. When LoRA disabled: [0].
    """
    if lora_config is None:
        return [0]
    if compilation_config.cudagraph_specialize_lora:
        specialize = getattr(lora_config, "specialize_active_lora", False)
        captured = get_captured_lora_counts(lora_config.max_loras, specialize)
        return [0] + [c for c in captured if c > 0]
    return [0, lora_config.max_loras + 1]

• Default: cudagraph_specialize_lora is on, specialize_active_lora is off (vllm/config/compilation.py:663-669, vllm/config/lora.py:66-72). So the cases are [0, max_loras + 1]: one graph set for batches without adapters, one for batches with any adapters. LoRA start-up captures twice as many graphs.
• With --specialize-active-lora: the cases are 0, the powers of 2 up to max_loras, and max_loras + 1 (vllm/lora/utils.py:55-69). A batch with 3 adapters and max_loras 4 uses the case 4 (vllm/v1/worker/gpu/cudagraph_utils.py:203-227).

The number of active adapters for the dispatch is the set of LoRARequests in the batch (vllm/v1/worker/gpu/lora_utils.py:38-49).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

What can go wrong#

• Requests wait with many adapters. More distinct adapters than --max-loras among running requests make new requests wait (Step 2).
• Latency spikes on new adapters. Loading stops the engine loop while it runs (Step 3). Preload the adapters.
• "LoRA rank ... is greater than max_lora_rank". Raise --max-lora-rank to the largest rank you serve (LoRA adapters).
• "... does not support LoRA yet." The model class does not implement SupportsLoRA (vllm/v1/worker/lora_model_runner_mixin.py:49-50).
• Unloaded adapters still use memory. Unload only removes the name in the API server (deep dive).
• Longer start-up. LoRA doubles the CUDA graph captures by default, and --specialize-active-lora adds more (Step 6).

Why it matters. Multi-LoRA serving puts many fine-tuned variants on one GPU. Its limits come from three fixed sizes: max_loras per step, max_cpu_loras in memory, and slot buffers sized for max_lora_rank.