Hugging Face checkpoints

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

When you run vllm serve Qwen/Qwen3-0.6B, the name points to a folder of files on the Hugging Face Hub, or to a local folder. vLLM needs four kinds of information from it. They are the shape of the model, the default sampling values, the tokenizer with its chat template, and the weights. Each kind lives in its own files.

config.json: the shape of the model#

The Hugging Face config is a JSON object. The keys that matter most for serving:

• architectures: the name of the model class, for example ["Qwen3ForCausalLM"]. vLLM looks this name up in its model registry to find its own implementation.
• max_position_embeddings: the longest context that the model was trained for. It becomes the default max_model_len.
• torch_dtype (in newer files dtype): the dtype of the weights, for example bfloat16. It becomes the default --dtype.
• quantization_config: only in quantized checkpoints (Quantized checkpoints).
• rope_scaling or rope_parameters: RoPE settings for long context. Some RoPE types make the usable context longer than max_position_embeddings.
• Sizes: hidden_size, num_hidden_layers, num_attention_heads, num_key_value_heads, vocab_size. They give the shapes of all weights and of the KV cache.

generation_config.json: default sampling values#

The generation config holds the values that the model authors suggest for sampling, for example temperature, top_p and top_k, and the end-of-sequence token ids. By default, vLLM uses these values when a request does not set them. So two models can sample differently with the same request.

Tokenizer files and the chat template#

The tokenizer turns text into token ids (Tokens↗). Its files are tokenizer.json (the vocabulary and merge rules) and tokenizer_config.json (special tokens and settings). The chat template turns a list of chat messages into one prompt. It is a Jinja template, in the key chat_template of tokenizer_config.json, or in its own file chat_template.jinja in newer repositories. Transformers reads both places (general knowledge about Transformers, not vLLM code).

Weights: safetensors files#

The weights are in safetensors files. A file starts with a JSON header that lists each tensor: name, dtype, shape and byte offsets. The raw bytes follow. Loading a safetensors file runs no code, unlike a pickled .bin file. A small model has one model.safetensors. A large model has shards such as model-00001-of-00004.safetensors, and model.safetensors.index.json maps each tensor name to its shard.

Where the files come from#

• A Hub name such as Qwen/Qwen3-0.6B. vLLM downloads the files with the huggingface_hub library into the Hugging Face cache. By default, this library uses ~/.cache/huggingface/hub. HF_HOME moves it to $HF_HOME/hub, and HF_HUB_CACHE sets it directly (general knowledge about huggingface_hub). A second start finds the files there.
• A local path such as /models/qwen3. vLLM reads the folder directly and downloads nothing.
• --revision selects a branch, a tag or a commit of the Hub repository. Pin a commit in production, so a new upload does not change your server.

An everyday comparison#

Think of a flat-pack furniture box. config.json is the label with the model name and the sizes. generation_config.json is the "recommended use" sheet. The tokenizer files are the dictionary for the instruction language. The safetensors files are the parts, in numbered bags, and the index is the parts list. vLLM is the worker who reads the label first, picks the right assembly plan (the model class), and then puts each part in its place.

The comparison stops here: vLLM does not use the assembly plan in the box (the Python code of the repository) by default. It uses its own plan for the architecture name. Only with --trust-remote-code can code from the repository run.

The picture#

Read panel 1 row by row, from the file on the left to what vLLM gets on the right. Panel 2 follows only the first row further: from the name on the command line to the class that the workers build.

In vLLM#

This is a basics page, so this part gives only pointers. The pages How a vLLM model file is written (llama.py), "Weight loading: from checkpoint files to GPU parameters" and Renderers and chat templates: from messages to token ids explain the code in full.

1. The default model. ModelConfig.model defaults to "Qwen/Qwen3-0.6B" (vllm/config/model.py:155), and EngineArgs copies it (vllm/engine/arg_utils.py:486). vllm serve <name> puts the positional name into args.model (vllm/entrypoints/cli/serve.py:56-57).
2. Name → config → class. ModelConfig.__post_init__() calls get_config() (vllm/config/model.py:680-690). It then asks the registry for the architecture: registry.inspect_model_cls(architectures, self) (vllm/config/model.py:739-744). On the workers, resolve_model_cls() imports the class (vllm/model_executor/model_loader/utils.py:310-313).
3. The registry entry. "Qwen3ForCausalLM": ("qwen3", "Qwen3ForCausalLM") maps the name to a module and a class (vllm/model_executor/models/registry.py:199).

        hf_config = get_config(
            self.hf_config_path or self.model,
            self.trust_remote_code,
            self._hf_config_revision or self.revision,
            self.code_revision,
            self.config_format,
            hf_overrides_kw=hf_overrides_kw,
            hf_overrides_fn=hf_overrides_fn,
            token=self.hf_token,
        )
        self.hf_config = hf_config

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

What can go wrong#

• "Invalid repository ID or local directory". The name is wrong, the repository is private (log in, or set HF_TOKEN), or the folder has no config.json (vllm/transformers_utils/config.py:904-917).
• Unexpected sampling. The model's generation_config.json changes the defaults, for example temperature 0.6 (Qwen3-0.6B on the Hub) instead of 1.0. Check the log line, or start with --generation-config vllm (vllm/config/model.py:1906-1914).
• A slow model on the Transformers fallback. If vLLM has no class for the architecture, it uses the Transformers implementation and warns (vllm/model_executor/model_loader/utils.py:315-323). Check the "Resolved architecture" log line (vllm/config/model.py:744).
• A model that changes under you. Without --revision, a new upload to the repository can change the weights or the chat template at the next start. Pin a commit.
• Missing shards. An index file that names a shard that is not in the folder raises FileNotFoundError (vllm/model_executor/model_loader/weight_utils.py:609-613).
• Remote code. Some repositories need --trust-remote-code. It runs Python code from the repository in every vLLM process. Read the code first, and pin the revision.

Why it matters. Most surprises at start-up come from the model folder: the wrong class, the wrong context length, other default sampling values or a missing file. When you know which file feeds which setting, you can fix them with the right flag.