Quantized checkpoints

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

BF16, FP8 and FP4 are number formats with fewer bits per value, and block scales map the small values back to real values (Number formats↗). A quantized checkpoint is a model folder that stores some tensors in such a small format. This page explains which tensors, how the checkpoint says it, and what that does to memory in a server.

Three families#

• Weight-only (W4A16, W8A16). Only the weights are small, for example 4-bit integers. The kernel expands them to 16 bits and multiplies with 16-bit activations. AWQ and GPTQ are the common 4-bit methods. This saves memory and memory bandwidth, so it helps decode, which reads all weights for each step. It does not make the multiply itself cheaper.
• Weight and activation (W8A8). Weights and the activations that enter each linear layer are 8-bit, FP8 or INT8. The GPU then runs the multiply in 8 bits. On GPUs with FP8 tensor cores (for example NVIDIA Hopper), the FP8 peak is about twice the BF16 peak. This is general knowledge, not from the vLLM repository. This helps prefill and large batches.
• FP4 (NVFP4, MXFP4). 4-bit floating point with a scale for each small block of values. On the newest GPUs, weights and activations can both be FP4 (W4A4). On other GPUs, vLLM can run some FP4 checkpoints weight-only.

Scales: per tensor, per channel, per group, per block#

A small format has a small range. So each group of values gets a scale: real value ≈ stored value × scale. The checkpoint stores the scales next to the weights. The group size is a trade-off: more scales give better accuracy but use more memory.

Activations change with every request. So their scales come from calibration data (static, one fixed scale), or from the live values (dynamic). A dynamic scheme computes, for example, one scale per token in the kernel.

The KV cache is separate#

KV cache quantization stores the keys and values in a small format, normally FP8. It is a separate choice from the weight format. You can serve a BF16 model with an FP8 KV cache, or an AWQ model with a BF16 KV cache. FP8 KV halves the bytes per token, so the same memory holds twice the tokens.

How a checkpoint says that it is quantized#

A Hugging Face model folder has a config.json (Hugging Face checkpoints). A quantized checkpoint adds a quantization_config object to it. The key quant_method names the format. The other keys depend on the format:

"quantization_config": {
  "quant_method": "awq",
  "bits": 4,
  "group_size": 128,
  "zero_point": true
}

Common values of quant_method are awq, gptq, fp8, compressed-tensors (from llm-compressor) and modelopt (from NVIDIA ModelOpt). A compressed-tensors config has config_groups. Each group says, for a list of target layers, how the weights and the input_activations are quantized.

Memory math: an 8B model#

Take a Llama-3.1-8B-shaped model: 8.03 billion weights. 6.98 billion are in the linear layers of the 32 decoder layers. 1.05 billion are in the embedding and the LM head, which many quantized checkpoints keep in BF16.

The assumptions: 1 GB = 10⁹ bytes, and the GPU has 24 GB. The model is untied, with a vocabulary of 128,256 tokens. The computation: the budget is 24 GB × 0.92 (the default gpu_memory_utilization, vllm/config/cache.py:212) = 22.1 GB. The KV cache needs 32 layers × 8 KV heads × 128 × 2 (K and V) × 2 bytes = 128 KiB per token in BF16. For FP8 weights, 22.1 − 9.1 = 13.0 GB, and 13.0 GB / 131,072 bytes = 99,200 tokens. The INT4 row counts one 16-bit scale and one 4-bit zero point per group of 128 weights, as AWQ stores them.

An everyday comparison#

Think of a photo library. Weight-only quantization is like small JPEG files that you open to full size when you look at them. They use less disk space, but the viewing work is the same. W8A8 is like a viewer that can show the small files directly, without the expand step. The KV cache is the stack of photos on your desk: if each one is smaller, more of them fit.

The comparison stops here: a JPEG loses detail in a fixed way. Quantization methods such as AWQ and GPTQ choose the scales with calibration data, to keep the error small where the model is sensitive.

The picture#

Read panel 1 row by row. The colour of a cell gives the number of bits. Panel 2 is the memory table of this page as a picture: the weight format moves the line between weight memory and KV cache memory.

In vLLM#

This is a basics page, so this part gives only pointers. The page Quantization methods as plugins explains how a quantization method plugs into the layers.

1. The flag. --quantization / -q selects a method (vllm/engine/arg_utils.py:985). Its default is None. Then vLLM reads quantization_config from the model config; without one, the weights are not quantized (vllm/config/model.py:264-268).
2. Detection. ModelConfig._verify_quantization() runs in __post_init__ (vllm/config/model.py:922). It reads quant_method and asks each method class whether it wants to take over the checkpoint (vllm/config/model.py:1388-1462).
3. The list. QuantizationMethods has 30 names: 24 methods such as awq, gptq, fp8, compressed-tensors, modelopt_fp4 and mxfp4, and 6 short names for online quantization such as fp8_per_tensor (vllm/model_executor/layers/quantization/__init__.py:15-48).
4. The KV cache. --kv-cache-dtype has the default "auto" (vllm/config/cache.py:232). A KV format in the checkpoint wins first; otherwise "auto" means the model dtype (next deep dives).

        # Parse quantization method from the HF model config, if available.
        quant_cfg = self.model_arch_config.quantization_config

        if quant_cfg is not None:
            quant_method = quant_cfg["quant_method"]
            # Quantization methods which are overrides (i.e. they have a
            # `override_quantization_method` method) must be checked in order
            # of preference (this is particularly important for GPTQ).
            overrides = [
                "auto_gptq",
                "gptq",
                "gptq_marlin",
                "auto_awq",
                "awq",
                "awq_marlin",
                "inc",
                "moe_wna16",
                "modelopt",
                "modelopt_fp4",
                "modelopt_mxfp8",
                "mxfp8",
                "modelopt_mixed",
                # Ensure heavy backends are probed last to avoid unnecessary
                # imports during override detection (e.g., MXFP4 imports Triton)
                "mxfp4",
                "gpt_oss_mxfp4",
                "deepseek_v4_fp8",
                "humming",
            ]
            # if the user specifies humming, we should always use humming
            if self.quantization == "humming":
                overrides = ["humming"] + overrides
            quantization_methods = [
                q for q in supported_quantization if q not in overrides
            ]
            # Any custom overrides will be in quantization_methods so we place
            # them at the start of the list so custom overrides have preference
            # over the built-in ones.
            quantization_methods = quantization_methods + overrides

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

vllm serve meta-llama/Llama-3.1-8B-Instruct --quantization fp8_per_tensor

What can go wrong#

• The flag does not match the checkpoint. --quantization fp8 on an AWQ checkpoint raises "Quantization method specified in the model config (awq) does not match ..." (vllm/config/model.py:1456-1462). Leave the flag out for a quantized checkpoint; vLLM detects the method.
• --quantization fp8 on a BF16 model. It still works, but it logs a deprecation warning and runs as fp8_per_tensor (vllm/model_executor/model_loader/weight_utils.py:356-363). Use fp8_per_tensor or another online short name.
• The GPU is too old. Each method has a minimum compute capability; vLLM stops at start-up below it (vllm/config/vllm.py:910-918).
• The dtype does not fit the method. Some methods support only FP16 or only BF16 activations. Start-up then fails with "is not supported for quantization method" (vllm/config/vllm.py:919-925). Set --dtype to a supported type.
• An FP8 KV cache without good scales. FP8 KV halves KV memory, but without calibrated scales the accuracy can drop. Test the quality of your task before you use it in production.
• Docs that show old flags. Guides from older versions use --quantization fp8 for online quantization. The code marks it as deprecated (vllm/model_executor/model_loader/weight_utils.py:357-360). Use the new short names.

Why it matters. The weight format decides how much GPU memory is left for the KV cache, and so how many requests the server can hold at once. The activation format decides how fast prefill and large batches run. Pick both on purpose, and keep the KV cache format as a separate decision.