LLM quantization explained
LLM quantization stores a model's weights, and sometimes activations or the KV cache, in fewer bits to cut memory, at the cost of some accuracy. Methods compared.
LLM quantization stores a model’s numbers in fewer bits. Hugging Face describes it as keeping a model’s weights, and sometimes its activations, in fewer bits so the model needs less memory. Fewer bits cost some accuracy, and each method trades memory, speed and accuracy differently. Quantization is one of the main levers for fitting an open-weight model onto the hardware you have.
What it saves
Models are usually loaded in bf16, which takes 2 bytes per parameter. Hugging Face’s overview works through an 8B model: about 16GB for its weights in bf16, about 8GB at 8 bits and about 4 to 5GB at 4 bits.
Those figures cover weights only. At long context or high batch sizes the KV cache can take a large share of memory too, which is why it can be quantized separately (see below).
How the mapping works
A quantized number can take only a few values: 256 for int8 and 16 for int4. The Hugging Face concept guide explains that every method therefore introduces rounding error, and methods differ in how they keep it from hurting the output.
The common mapping is affine. It stretches the integer range over a tensor’s minimum and maximum using a scale and a zero point, and dequantizing gives back an approximation of the original value. Symmetric quantization fixes the zero point at 0, which is cheaper but can leave some values unused.
How many values share one scale matters. The guide lists three choices:
| Granularity | Scales | Trade-off |
|---|---|---|
| Per-tensor | One for the whole tensor | Cheap, but one large outlier stretches the range for every other weight |
| Per-channel | One per output channel (a row of the weight matrix) | Narrower range per scale |
| Per-group | One per group of consecutive values, such as 128 weights | Best range fit, with extra storage for each scale |
Most low-bit methods quantize per group, and outliers are why. Dettmers and colleagues found that large models develop systematic outlier features that dominate attention. Their LLM.int8() method runs those dimensions in 16-bit and keeps over 99.9% of values in 8-bit.
Weights, activations and the KV cache
| What is quantized | Examples | When it helps |
|---|---|---|
| Weights only | bitsandbytes 4-bit, AWQ, GPTQ, HQQ | Memory-bound decoding. The kernel dequantizes weights before each matmul. |
| Weights and activations (W8A8) | FP8 and INT8 schemes | Compute-bound work such as prefill and large batches. Needs hardware support for the low-precision matmul. |
| KV cache | Hugging Face QuantizedCache |
Long contexts where the cache dominates memory |
The concept guide explains why weight-only methods do not always run faster. Generating one token at a time, a GPU spends most of each step reading weights from memory, so 4-bit weights move a quarter of the data bf16 weights do. That can outweigh the cost of dequantizing.
For the cache, Hugging Face’s docs describe a quantized cache with hqq, quanto and fp8 backends. They warn it can harm latency when the context is short and GPU memory is enough without it.
Data types
| Type | Notes | Source |
|---|---|---|
| int8, int4 | Evenly spaced integers; int4 has only 16 values, so it loses more accuracy | Concept guide |
| FP8 (E4M3, E5M2) | Floating point in 8 bits; E4M3 is the usual inference choice; matmuls need NVIDIA compute capability 8.9 or higher, AMD MI300 or Intel XPU | Concept guide |
| NF4 | 4-bit type with more values near zero, where most weights are | Concept guide; QLoRA paper |
| MXFP4, NVFP4 | 4-bit floats with a scale per small block: 32 values and a power-of-two scale for MXFP4, 16 values and an FP8 scale (plus one FP32 per-tensor scale) for NVFP4 | Concept guide |
| Codebooks (AQLM, VPTQ) | Groups of weights stored as an index into learned vectors; used below 4 bits | Concept guide |
When quantization happens
Post-training quantization (PTQ) works on a model that is already trained. Some PTQ methods need no data: the scales come from the weights, so quantization is fast enough to run as the model loads, as with bitsandbytes, HQQ and torchao. Others calibrate on a small dataset, which usually gives better accuracy at 4 bits and below but takes minutes to hours.
Two calibrated methods are widely used:
- GPTQ. Frantar and colleagues describe a one-shot method that uses approximate second-order information. It quantized a 175B-parameter model in about four GPU hours, to 3 or 4 bits per weight with negligible accuracy loss, and reported end-to-end speedups of about 3.25x on an A100 and 4.5x on an A6000 over FP16.
- AWQ. Lin and colleagues report that protecting only 1% of salient weights can greatly reduce quantization error, with those channels found from the activation distribution and no backpropagation or reconstruction.
Quantization-aware training (QAT) simulates the rounding during training so the model learns to tolerate it. The concept guide says it usually gives the best accuracy at very low bit widths but requires training the model. BitNet’s 1.58-bit weights are trained this way.
Choosing a method
Hugging Face’s method guide recommends a starting point per goal. The table below follows it as of November 2026.
| Goal | Start with |
|---|---|
| Fit a model in less memory with the least setup | bitsandbytes (4 or 8 bits, no calibration) |
| Fine-tune with QLoRA | bitsandbytes with NF4 |
Speed up inference with torch.compile |
torchao |
| Best 4-bit accuracy | GPTQ, or AWQ and AutoRound quantized with their own libraries |
| FP8 on Ada, Hopper or Intel XPU | Fine-grained FP8 |
| FP4 on Blackwell | NVFP4 |
| Below 4 bits | HQQ or SINQ, or AQLM, VPTQ and SpQR checkpoints |
If a checkpoint is already quantized, the guide says the checkpoint decides the method. Most GGUF models, the overview adds, are dequantized to full precision as they load in Transformers, so they use as much memory as an unquantized model there.
How much accuracy you lose
Hugging Face’s benchmarks on Llama 3.1 8B and 70B group the results by bit width:
| Bits | Memory vs bf16 | Accuracy |
|---|---|---|
| 8 | About 2x smaller | Very close to bf16 |
| 4 | About 4x smaller | High; AWQ and GPTQ often lead but need calibration |
| Below 4 | More than 4x smaller | Noticeable drop, especially at 2 bits |
An independent study agrees on the shape. Kurtic and colleagues ran over 500,000 evaluations on Llama-3.1 and found FP8 W8A8 “effectively lossless across all model scales,” well-tuned INT8 W8A8 at 1 to 3% degradation, and INT4 weight-only “more competitive than expected, rivaling 8-bit quantization.” They recommend W4A16 for cost efficiency in synchronous serving and W8A8 for asynchronous continuous batching. Both studies cover Llama models, so check your own.
Test the quantized model on your task
Benchmark averages hide task-level drops. Hugging Face’s guide tells readers to benchmark accuracy and speed on their own task and hardware before choosing a method for production. In practice that means running the same evaluation set against the full-precision and quantized versions, and keeping it as a regression test when you swap checkpoints.
HoneyHive’s tracing concepts describe events that store model inputs, outputs and metadata such as token counts and cost, so you can compare latency, cost and outputs per call between two deployments. For the serving side, see LLM inference and LLM cost optimization.
FAQ
What is quantization in an LLM?
It stores a model's weights, and sometimes its activations, in fewer bits so the model needs less memory. Hugging Face's docs note that fewer bits cost some accuracy, and each method makes a different trade-off between memory, speed and accuracy.
How much memory does quantization save?
Hugging Face's overview gives an 8B model as about 16GB of weights in bf16, about 8GB at 8 bits and about 4 to 5GB at 4 bits. That counts weights only, not the KV cache.
Does quantization hurt accuracy?
A little at 8 bits and 4 bits, more below that. Kurtic and colleagues evaluated Llama-3.1 models and found FP8 effectively lossless, well-tuned INT8 at 1 to 3% degradation, and 4-bit weight-only competitive with 8-bit. Hugging Face's benchmarks show a noticeable drop below 4 bits.
What is the difference between GPTQ and AWQ?
Both are post-training, weight-only methods that calibrate on data. The Hugging Face guide says GPTQ adjusts the remaining columns to make up for each column's rounding error, while AWQ scales up the channels with the largest activations before rounding.
Is quantization the same as fine-tuning?
No. Post-training quantization changes how weights are stored, not what the model learned. QLoRA combines the two: the paper freezes a 4-bit base model and trains small LoRA adapters on top.