"Weight loading: from checkpoint files to GPU parameters"

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

A checkpoint is a set of named tensors in safetensors files, and loading copies each tensor into the matching model parameter (Weight loading↗). The page Hugging Face checkpoints shows which files vLLM downloads and keeps. This page follows the tensors from the files into the parameters of each GPU.

vLLM loads the weights in six steps, in every worker:

1. Pick a loader. The load format (--load-format, default auto) selects a model loader class. For auto it is DefaultModelLoader.
2. Build an empty model. The loader builds the model class on the GPU. Each tensor-parallel layer makes parameters of its per-rank shape, with no values yet.
3. Stream the tensors. An iterator opens the files one by one and gives (name, tensor) pairs. In the default path, the tensors stay on the CPU.
4. Map the names. The model's load_weights() uses an AutoWeightsLoader. A WeightsMapper renames checkpoint names to vLLM names, for example q_proj to qkv_proj with the shard id "q".
5. Copy the slice. The target parameter has a weight loader. It cuts the part of the tensor that this rank owns and copies it into the right rows of the parameter.
6. Post-process. Each layer's quantization method can repack or quantize its weights once, in process_weights_after_loading.

An everyday comparison. Think of the branches of a library that all get the same delivery list. Each branch reads every line of the list. For each book, the list gives the shelf name in the publisher's words, and a branch translates it into its own shelf names. Some books go into a shared box with three marked parts, such as Q, K and V. Each branch keeps only the pages for its own readers. The comparison stops here. In vLLM, nobody sends a branch its part: every GPU walks the full list itself, and the parameter cuts its slice.

The picture#

Read panels 1 and 2 from left to right: they run the same way in every worker. Panel 3 shows the only step where the ranks differ. The rows of the stacked parameter are the same on both ranks; the rows taken from the checkpoint tensor depend on the rank.

In the code#

Step 1: the load format picks the loader#

_LOAD_FORMAT_TO_MODEL_LOADER: dict[str, type[BaseModelLoader]] = {
    "auto": DefaultModelLoader,
    "hf": DefaultModelLoader,
    "dummy": DummyModelLoader,
    "fastsafetensors": DefaultModelLoader,
    "instanttensor": DefaultModelLoader,
    "ipc_cache": IpcModelLoader,
    "mistral": DefaultModelLoader,
    "modelexpress": ModelExpressModelLoader,
    "npcache": DefaultModelLoader,
    "pt": DefaultModelLoader,
    "runai_streamer": RunaiModelStreamerLoader,
    "runai_streamer_sharded": ShardedStateLoader,
    "safetensors": DefaultModelLoader,
    "sharded_state": ShardedStateLoader,
    "tensorizer": TensorizerLoader,
}

get_model_loader() looks up load_config.load_format in this table and raises ValueError for an unknown name (vllm/model_executor/model_loader/__init__.py:123-128). The field default is "auto", and a validator lower-cases the value (vllm/config/load.py:30, vllm/config/load.py:136-138). EngineArgs copies this default without a change (vllm/engine/arg_utils.py:509). One later override exists: for a model in object storage (S3, GCS, Azure), auto becomes runai_streamer (vllm/config/vllm.py:2754-2762). A plugin can add a format with register_model_loader (Hooks in the engine core and the workers).

Model Runner V2 calls the loader in load_model() (vllm/v1/worker/gpu/model_runner.py:398-406). It runs in every worker: with TP = 2, two processes do all the steps below at the same time.

Step 2: build the model with empty parameters#

    @instrument(span_name="Load model")
    def load_model(
        self, vllm_config: VllmConfig, model_config: ModelConfig, prefix: str = ""
    ) -> nn.Module:
        """Load a model with the given configurations."""
        device_config = vllm_config.device_config
        load_config = vllm_config.load_config
        load_device = (
            device_config.device if load_config.device is None else load_config.device
        )
        target_device = torch.device(load_device)
        with set_default_torch_dtype(model_config.dtype):
            with target_device:
                time_before_load = time.perf_counter()
                model = self.create_model(
                    vllm_config=vllm_config,
                    model_config=model_config,
                    prefix=prefix,
                )
                time_after_load = time.perf_counter()
                logger.info_once(
                    "Initializing model took %.6f seconds",
                    time_after_load - time_before_load,
                )

            logger.debug("Loading weights on %s ...", load_device)
            self.load_weights(model, model_config)

            # Release transient tensors held by PyTorch's caching allocator
            # before weight post-processing and raw CUDA driver allocations.
            gc.collect()
            torch.accelerator.empty_cache()

            # Log peak GPU memory after loading weights. This is needed
            # to have test coverage on peak memory for online quantization.
            if current_platform.is_cuda_alike() or current_platform.is_xpu():
                peak_memory = torch.accelerator.max_memory_allocated()
                logger.debug_once(
                    "Peak GPU memory after loading weights: %s GiB",
                    format_gib(peak_memory),
                )

            # Process weights into kernel format. Note that when using online
            # quantization, weights are (typically) quantized as they are loaded.
            if _has_online_quant(model):
                finalize_layerwise_processing(model, model_config)

            process_weights_after_loading(model, model_config, target_device)
            log_online_quantization_time(vllm_config)

            gc.collect()
            torch.accelerator.empty_cache()

        return model.eval()

1. Lines 73-74. The default dtype is the model dtype, and the default device is the GPU of this worker. So every torch.empty in a layer constructor makes a GPU tensor of the right dtype.
2. Lines 76-80. create_model() calls initialize_model(), which builds the class with vllm_config and prefix (How a vLLM model file is written (llama.py)). The log line "Initializing model took ... seconds" measures this part.
3. Line 88. self.load_weights() is the loader's method; for the default loader it streams the files (Step 3).
4. Lines 106-109. Online quantization finishes first; then process_weights_after_loading() runs for every layer (Step 6).

The layer constructors do the per-rank sizing. UnquantizedLinearMethod.create_weights() makes torch.empty(sum(output_partition_sizes), input_size_per_partition) (vllm/model_executor/layers/linear.py:178-205). The page The tensor-parallel layers (linear.py and vocab_parallel_embedding.py) gives these sizes for each layer class.

Step 3: choose the files and stream the tensors#

DefaultModelLoader._prepare_weights() finds, downloads and filters the files (vllm/model_executor/model_loader/default_loader.py:147-263); Hugging Face checkpoints explains that part. Then _get_weights_iterator() picks an iterator by format (vllm/model_executor/model_loader/default_loader.py:301-365). The usual case is safetensors_weights_iterator():

        else:
            with safe_open(st_file, framework="pt") as f:
                for name in f.keys():  # noqa: SIM118
                    if should_skip_weight(name, local_expert_ids):
                        continue
                    param = f.get_tensor(name)
                    yield name, param

• Order. The files are sorted by a natural sort key, so ...-00002-of-... comes before ...-00010-of-... (vllm/model_executor/model_loader/weight_utils.py:1134).
• Strategies. --safetensors-load-strategy is None by default. eager reads the whole file into CPU memory first; torchao rebuilds torchao tensors (vllm/model_executor/model_loader/weight_utils.py:1207-1242).
• Prefetch. With the default None, vLLM prefetches the files into the page cache only on NFS or Lustre. They must also fit in 90% of the free RAM (vllm/model_executor/model_loader/weight_utils.py:1136-1155). prefetch forces it. Each process prefetches a different part of the files, sorted_files[rank::world_size]. Rank and world size are the global values of torch.distributed, not the TP values (vllm/model_executor/model_loader/weight_utils.py:1051-1057).
• PyTorch files. For .bin and .pt files, pt_weights_iterator() calls torch.load(..., weights_only=True) on each file and yields its items (vllm/model_executor/model_loader/weight_utils.py:1479-1495).
• Many threads. --model-loader-extra-config '{"enable_multithread_load": true}' selects the multi-thread iterators, with 8 threads by default (vllm/model_executor/model_loader/default_loader.py:58-59, vllm/model_executor/model_loader/default_loader.py:329-336).

The loader then calls model.load_weights() with the iterator, and logs "Loading weights took ... seconds" (vllm/model_executor/model_loader/default_loader.py:503-509).

Step 4: map the names (WeightsMapper and AutoWeightsLoader)#

Llama has two load_weights methods. The outer class makes an AutoWeightsLoader without a mapper (vllm/model_executor/models/llama.py:559-561). The loader groups the names by their first part and finds the child model. That child has its own load_weights, so the loader gives it the names without model. (vllm/model_executor/models/utils.py:387-400). LlamaModel.load_weights uses the mapper:

    hf_to_vllm_mapper = WeightsMapper(
        orig_to_new_stacked={
            # weight_name: (param_name, shard_id)
            ".q_proj": (".qkv_proj", "q"),
            ".k_proj": (".qkv_proj", "k"),
            ".v_proj": (".qkv_proj", "v"),
            ".gate_proj": (".gate_up_proj", 0),
            ".up_proj": (".gate_up_proj", 1),
        }
    )

        shard_id: ShardIds | None = None
        for substr, (new_key, new_shard_id) in self.orig_to_new_stacked.items():
            if substr in key:
                key = key.replace(substr, new_key, 1)
                shard_id = new_shard_id

    def apply(
        self, weights: Iterable[tuple[str, torch.Tensor]]
    ) -> Iterable[tuple[str, torch.Tensor]]:
        for name, data in weights:
            result = self._map_name_with_shard(name)
            if result is None:
                continue
            out_name, shard_id = result
            if shard_id is None:
                yield out_name, data
                continue
            shard_ids = shard_id if isinstance(shard_id, list) else [shard_id]
            for i, one_shard_id in enumerate(shard_ids):
                # Each shard carries its own id, so extra shards need their own
                # tensor object; `detach` aliases the storage instead of copying.
                shard_data = data if i == 0 else data.detach()
                shard_data.shard_id = one_shard_id
                yield out_name, shard_data

The mapper has six kinds of rule: orig_to_new_renaming, _regex, _substr, _stacked, _prefix and _suffix, applied in this order. A regex, substring, prefix or suffix rule that maps to None drops the tensor (vllm/model_executor/models/utils.py:48-61, vllm/model_executor/models/utils.py:108-145). AutoWeightsLoader.load_weights() adds more mappers. With a quantization config, it adds the config's mappers for KV cache scale names and checkpoint names. It always adds a rule that drops old rotary tables (vllm/model_executor/models/utils.py:460-473).

Then the loader walks the module tree:

        child_modules = dict(module.named_children())
        child_params = dict(module.named_parameters(recurse=False))

        # Add missing tensors the weight loader needs to be able to load
        # that aren't registered as params, e.g., batchnorm statistics.
        self._add_loadable_non_param_tensors(module, child_params)

        for child_prefix, child_weights in self._groupby_prefix(weights):
            prefix = self._get_qualname(base_prefix, child_prefix)

            if child_prefix in child_modules:
                yield from self._load_module(
                    prefix, child_modules[child_prefix], child_weights
                )
            elif child_prefix in child_params:
                if self._can_skip(prefix):
                    logger.debug("Skipping param %s", prefix)

                    continue

                yield from self._load_param(
                    prefix, child_params[child_prefix], child_weights
                )

1. Line 409. _groupby_prefix() splits each name at the first dot: layers.0.self_attn.qkv_proj.weight becomes layers and 0.self_attn.qkv_proj.weight.
2. Lines 412-415. A child module: recurse. For QKVParallelLinear the recursion ends early, because that layer has its own load_weights (line 388).
3. Lines 416-424. A parameter: _load_param() calls param.weight_loader, or default_weight_loader when the parameter has none (vllm/model_executor/models/utils.py:340-341).

QKVParallelLinear.load_weights reads the shard id from the tensor and calls param.weight_loader(param, loaded_weight, shard_id) (vllm/model_executor/layers/linear.py:1339-1354). MergedColumnParallelLinear has the same method for the integer shard ids of gate_up_proj (vllm/model_executor/layers/linear.py:985-1000).

Step 5: the weight loader of each parameter#

A linear layer chooses one of two loader families when it is built. If the class name of its quantization method is in WEIGHT_LOADER_V2_SUPPORTED, it uses weight_loader_v2; else weight_loader (vllm/model_executor/layers/linear.py:530-534). UnquantizedLinearMethod is in the list, so a BF16 model uses v2 (vllm/model_executor/layers/linear.py:50-64). The v2 loaders give the work to the parameter class:

    def load_column_parallel_weight(self, loaded_weight: torch.Tensor):
        shard_size = self.data.shape[self.output_dim]
        loaded_weight = loaded_weight.narrow(
            self.output_dim, self.tp_rank * shard_size, shard_size
        )
        assert self.data.shape == loaded_weight.shape
        self.data.copy_(loaded_weight)

    def load_row_parallel_weight(self, loaded_weight: torch.Tensor):
        shard_size = self.data.shape[self.input_dim]
        loaded_weight = loaded_weight.narrow(
            self.input_dim, self.tp_rank * shard_size, shard_size
        )

        if len(loaded_weight.shape) == 0:
            loaded_weight = loaded_weight.reshape(1)

        assert self.data.shape == loaded_weight.shape
        self.data.copy_(loaded_weight)

The shard size is the size of the parameter on this rank, so the loader needs no TP arithmetic of its own. ModelWeightParameter inherits both methods, and the layer calls the one that fits (vllm/model_executor/parameter.py:229-234). The v1 loaders do the same cut with getattr(param, "output_dim") and "input_dim" (vllm/model_executor/layers/linear.py:589-606, vllm/model_executor/layers/linear.py:1758-1774).

For a stacked parameter, the loader first selects the part of the parameter, then the part of the tensor:

    def load_qkv_weight(self, loaded_weight: torch.Tensor, **kwargs):
        shard_offset: int = kwargs["shard_offset"]
        shard_size: int = kwargs["shard_size"]
        shard_id: str = kwargs["shard_id"]
        num_heads: int = kwargs["num_heads"]

        # TODO: move these to PackedColumnParameter and PackedvLLMParameter
        if (
            isinstance(self, (PackedColumnParameter, PackedvLLMParameter))
            and self.output_dim == self.packed_dim
        ):
            shard_size, shard_offset = self.adjust_shard_indexes_for_packing(
                shard_offset=shard_offset, shard_size=shard_size
            )

        param_data = self.data
        shard_id_int = self.tp_rank if shard_id == "q" else self.tp_rank // num_heads
        param_data = param_data.narrow(self.output_dim, shard_offset, shard_size)
        loaded_weight = loaded_weight.narrow(
            self.output_dim, shard_id_int * shard_size, shard_size
        )

        assert param_data.shape == loaded_weight.shape
        param_data.copy_(loaded_weight)

1. Line 191. For q, the slice index is the TP rank. For k and v it is tp_rank // num_kv_head_replicas; num_heads here carries the replica count (vllm/model_executor/layers/linear.py:1203-1209).
2. Line 192. The part of the parameter: offset and size from the head counts of this rank (vllm/model_executor/layers/linear.py:1101-1117).
3. Lines 193-195. The part of the checkpoint tensor. Then an assertion and one copy.

Other parameters:

• Embedding and LM head. VocabParallelEmbedding.weight_loader cuts rows org_vocab_start_index to org_vocab_end_index, and fills the padding rows with zeros (vllm/model_executor/layers/vocab_parallel_embedding.py:497-520).
• Norm weights. RMSNorm.weight is a plain nn.Parameter without a loader (vllm/model_executor/layers/layernorm.py:65-67). default_weight_loader checks the shape and copies the full tensor on every rank (vllm/model_executor/model_loader/weight_utils.py:1529-1547).

Step 6: older model files: stacked_params_mapping#

Before WeightsMapper had stacked rules, each model wrote the loop itself. 38 files in vllm/model_executor/models/ still do. 98 other files use orig_to_new_stacked, and no file uses both (git grep at this commit). An example:

    def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
        stacked_params_mapping = [
            # (param_name, shard_name, shard_id)
            ("qkv_proj", "q_proj", "q"),
            ("qkv_proj", "k_proj", "k"),
            ("qkv_proj", "v_proj", "v"),
            ("gate_up_proj", "gate_proj", 0),
            ("gate_up_proj", "up_proj", 1),
        ]
        expert_params_mapping = [
            # (param_name, weight_name, expert_id)
            (
                "ws" if weight_name in ["w1", "w3"] else "w2s",
                f"experts.{expert_id}.{weight_name}.weight",
                expert_id,
            )
            for expert_id in range(self.num_experts)
            for weight_name in ["w1", "w2", "w3"]
        ]
        params_dict = dict(self.named_parameters())

        loaded_params: set[str] = set()
        for name, loaded_weight in weights:
            if "rotary_emb.inv_freq" in name:
                continue
            if "rotary_emb.cos_cached" in name or "rotary_emb.sin_cached" in name:
                # Models trained using ColossalAI may include these tensors in
                # the checkpoint. Skip them.
                continue
            for param_name, weight_name, shard_id in stacked_params_mapping:
                if weight_name not in name:
                    continue
                name = name.replace(weight_name, param_name)
                # Skip loading extra bias for GPTQ models.
                if name.endswith(".bias") and name not in params_dict:
                    continue
                if is_pp_missing_parameter(name, self):
                    continue
                param = params_dict[name]
                weight_loader = param.weight_loader
                weight_loader(param, loaded_weight, shard_id)
                break

1. Lines 489-496. The list has the same content as Llama's mapper, as tuples (param_name, shard_name, shard_id).
2. Lines 517-528. For each name, find a matching tuple and rename. Skip parameters of other pipeline stages. Call the loader with the shard id as the third argument.

Step 7: quantized weights#

For a quantized checkpoint, the layer's quantization method makes the parameters in create_weights() (Quantization methods as plugins). FP8, for example, registers an FP8 weight and a weight_scale (or weight_scale_inv for block scales), each with the layer's weight loader (vllm/model_executor/layers/quantization/fp8.py:312-339). The checkpoint has tensors with the same names, so the mapping and slicing above work for them too. A per-tensor scale of a stacked layer has one slot per part (q, k, v). PerTensorScaleParameter copies each checkpoint scale into the slot of its shard id (vllm/model_executor/parameter.py:253-301); the v1 loaders use adjust_scalar_to_fused_array for the same job (vllm/model_executor/layers/linear.py:97-121).

After all tensors are copied, process_weights_after_loading() visits every module with a quantization method and calls its post-processing (vllm/model_executor/model_loader/utils.py:157-196). FP8 with per-tensor scales, for example, re-quantizes the parts of a stacked layer to one scale, the maximum of their scales, and transposes the weight. This happens only without block scales and without the Marlin kernel (vllm/model_executor/layers/quantization/fp8.py:372-409, vllm/model_executor/layers/quantization/utils/fp8_utils.py:1454-1479). The page Quantized checkpoints explains the formats.

Step 8: the other loaders#

The table maps 15 format names to 7 classes (vllm/model_executor/model_loader/__init__.py:50-66). There is no gguf and no bitsandbytes format at this commit.

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

    @support_quantized_model_reload_from_hp_weights
    def load_weights(
        self,
        weights: Iterable[tuple[str, torch.Tensor]],
        *,
        mapper: WeightsMapper | None = None,
    ) -> set[str]:
        # Ignore unexpected biases (typically from GPTQ models)
        self.ignore_unexpected_suffixes.append(".bias")

        mapper = mapper or WeightsMapper()
        # Many models store quant_config in the base model instead of the causal model.
        # We look at the causal model's direct children for this reason.
        modules = (self.module, *self.module.children())
        iterator = (m.quant_config for m in modules if hasattr(m, "quant_config"))
        if quant_config := next(iterator, None):
            # Apply mappings for quantization-specific checkpoint tensors.
            mapper |= quant_config.get_cache_scale_mapper()
            mapper |= quant_config.get_checkpoint_weight_mapper()
            ignore_unexpected_suffixes = quant_config._ignore_unexpected_suffixes
            self.ignore_unexpected_suffixes.extend(ignore_unexpected_suffixes)
        mapper |= self.REMOVE_UNUSED_ROTARY_EMBEDS_MAPPER

        weights = mapper.apply(weights)
        weights = self._filter_skipped(weights)

        autoloaded_weights = set(self._load_module("", self.module, weights))
        self._check_skipped_aliases(autoloaded_weights)
        return autoloaded_weights

What can go wrong#

• A shape mismatch. The weight loaders assert that the slice and the parameter have the same shape (vllm/model_executor/parameter.py:197, vllm/model_executor/model_loader/weight_utils.py:1538-1541). A checkpoint for another config, or a wrong --hf-overrides, fails here.
• Missing weights. An unquantized model raises "Following weights were not initialized from checkpoint" for an empty norm weight, or another parameter outside quantize-method modules (vllm/model_executor/model_loader/default_loader.py:533-572). A missing linear weight gives no error here (derived).
• A dtype mismatch. A tensor with another dtype is converted by copy_, and the loader logs a warning (vllm/model_executor/models/utils.py:324-339).
• An unknown format. --load-format gguf raises "Load format `gguf` is not supported" (vllm/model_executor/model_loader/__init__.py:126-127).
• Slow loads from network storage. On NFS, a large model that does not fit in 90% of the free RAM is not prefetched, and vLLM logs a warning (vllm/model_executor/model_loader/weight_utils.py:1156-1165). Try --safetensors-load-strategy eager or prefetch, or a pre-sharded checkpoint.

Why it matters. Weight loading is often the longest part of start-up, and its errors are the first errors of a new model or a new checkpoint. When you know the path from file to parameter, you can read the log timers, fix name and shape errors, and choose a faster format.