LoRA adapters

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

What a LoRA adapter is#

Fine-tuning changes the weights of a model for one task. Full fine-tuning changes all weights, so each task gets a full copy of the model: 16 GB for an 8B model in BF16. LoRA (low-rank adaptation) instead keeps the base weight W frozen and learns a small update for it:

W' = W + (alpha / r) · B · A        W: [d_out, d_in]   B: [d_out, r]   A: [r, d_in]

The inner size r is the LoRA rank, often 8 to 64. Take a 4,096 × 4,096 projection and r = 16. The full matrix has 16.8 million values, but A and B together have only 131,072. A LoRA adapter is the set of these A and B matrices for some layers, together with adapter_config.json. That file gives r, lora_alpha and target_modules, the layers that get an update, for example q_proj, k_proj, v_proj, o_proj.

Merged or unmerged#

• Merged. You compute W' once and save a new full model. It runs at the full speed of the base model, but each adapter is a full model copy, and one server serves one adapter.
• Unmerged. The server keeps W and the small A and B apart. For each token it computes x W + (alpha / r) · (x A) B. The extra work is small, because r is small. One server can now keep one base model and many adapters.

vLLM serves adapters unmerged. This makes multi-LoRA serving serving possible: requests for different adapters run in the same batch.

Many adapters in one batch#

Batching reads the base weights once for the whole batch (Batching↗). A naive multi-adapter server would split the batch by adapter and lose this. The Punica approach keeps one batch:

1. Run the base matrix multiply x W for all tokens of the batch together.
2. Give each token the adapter id of its request.
3. Run batched LoRA kernels. Each token multiplies with the A and B of its own adapter. The "shrink" gives r values, the "expand" goes back to d_out, and the result is added.

So a batch with ten requests for five adapters still reads W only once. The adapters must be in GPU memory, in a fixed number of slots.

Slots and limits#

• max_loras: the number of GPU slots. It is the maximum number of different adapters in one batch.
• max_lora_rank: the largest rank that a slot can hold. vLLM sizes every slot for this rank, so a larger value uses more GPU memory for all slots.
• max_cpu_loras: the number of adapters to keep in CPU memory, ready to move into a free GPU slot.

Adapter memory: worked numbers#

Take a Llama-3.1-8B-shaped model: hidden size 4,096, 32 layers, intermediate size 14,336. It has 8 KV heads of size 128, so K and V have 1,024 outputs. An adapter with r = 16 on q_proj, k_proj, v_proj and o_proj has, per layer:

For 32 layers: 425,984 × 32 = 13.6 million values = 27.3 MB in BF16. That is 0.17 % of the 16.1 GB base model. So a GPU can hold many adapters next to one base model. The adapter files load quickly, too.

Add the MLP layers too: gate, up and down give 3 × 16 × (4,096 + 14,336) = 884,736 values per layer. The adapter then has 41.9 million values, or 83.9 MB.

An everyday comparison#

Think of one large printing press and a set of thin stencils. The press (the base model) prints every page. Each customer brings a stencil (an adapter) that changes a few details. The press prints pages for several customers in one run, and each page gets the stencil of its customer. The rack next to the press has a fixed number of stencil holders (the GPU slots).

The comparison stops here: a stencil only covers a part of a page. A LoRA adapter changes the output of every layer that it targets, and these changes add up through the whole model.

The picture#

Read panel 1 from left to right: one cell per scheduled token, coloured by adapter. Panel 2 is the computation of one LoRA-capable linear layer for these 7 tokens. The base weight W appears once; only the small stacked A and B tensors depend on the adapter.

In vLLM#

This is a basics page, so this part gives only pointers. The page Serving LoRA adapters: from the model name to the wrapped layer explains the LoRA layers, the managers and the kernels.

1. Turn it on. --enable-lora (default off) makes vLLM build a LoRAConfig; without it, there is no LoRA config (vllm/engine/arg_utils.py:688-693, vllm/engine/arg_utils.py:2680-2698). The defaults are max_loras 1 and max_lora_rank 16 (vllm/config/lora.py:33-36). max_cpu_loras unset means max_loras (vllm/config/lora.py:119-126).
2. Name the adapters. --lora-modules name=path or a JSON object registers adapters at start-up (vllm/entrypoints/common/cli_args.py:50-68). The API server loads them before it serves (vllm/entrypoints/launchers/api_server/app_state.py:86, vllm/entrypoints/openai/models/serving.py:124-139).
3. Select an adapter per request. The model field of a request names the base model or an adapter. _maybe_get_adapters() returns the LoRARequest of that name (vllm/entrypoints/serve/engine/serving.py:194-200).

vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-lora \
  --max-loras 4 --max-lora-rank 16 \
  --lora-modules sql=/adapters/sql-lora chat-de=/adapters/german-lora

curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model": "sql", "messages": [{"role": "user", "content": "Count the users per country."}]}'

    def _maybe_get_adapters(
        self,
        request: AnyRequest | AnyPoolingRequest,
        supports_default_mm_loras: bool = False,
    ) -> LoRARequest | None:
        if request.model in self.models.lora_requests:
            return self.models.lora_requests[request.model]

        # Currently only support default modality specific loras
        # if we have exactly one lora matched on the request.
        if supports_default_mm_loras:
            default_mm_lora = self._get_active_default_mm_loras(request)
            if default_mm_lora is not None:
                return default_mm_lora

        if self._is_model_supported(request.model):
            return None

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

curl -X POST localhost:8000/v1/load_lora_adapter -H "Content-Type: application/json" \
  -d '{"lora_name": "legal", "lora_path": "/adapters/legal-lora"}'
curl -X POST localhost:8000/v1/unload_lora_adapter -H "Content-Type: application/json" \
  -d '{"lora_name": "legal"}'

What can go wrong#

• "The model `x` does not exist" (HTTP 404). The model field names neither the base model nor a loaded adapter (vllm/entrypoints/serve/engine/serving.py:75-84). GET /v1/models lists the base model and every loaded adapter (vllm/entrypoints/openai/models/serving.py:149-165).
• Rank too large. An adapter with r = 32 fails with the default --max-lora-rank 16 (vllm/lora/peft_helper.py:194-197). Set the flag to the largest rank that you serve, but not higher: every slot reserves memory for it.
• Requests wait with many adapters. With the default --max-loras 1, a batch has only one adapter (vllm/v1/core/sched/scheduler.py:1001-1008). Requests for other adapters wait. Raise --max-loras to the number of adapters that run at the same time.
• Less KV cache. The LoRA buffers take memory before the KV cache gets its share. Large max_loras × max_lora_rank values reduce the number of tokens in the KV cache. --lora-target-modules can remove the buffers of modules that your adapters do not use.
• Runtime loading with several API servers. Start-up fails (vllm/entrypoints/cli/serve.py:315-318). Use --lora-modules or one API server.

Why it matters. Multi-LoRA serving lets one GPU serve many fine-tuned variants for the price of one base model. To get this, size max_loras and max_lora_rank for your real traffic.