# LLM quantization explained

> LLM quantization stores a model's weights, and sometimes activations or the KV cache, in fewer bits to cut memory, at the cost of some accuracy. Methods compared.

Source: https://www.honeyhive.ai/learn/llm-quantization (updated 2026-11-19)

LLM quantization stores a model's numbers in fewer bits. Hugging Face [describes](https://huggingface.co/docs/transformers/main/en/quantization/overview) it as keeping a model's weights, and sometimes its activations, in fewer bits so the model needs less memory. Fewer bits cost some accuracy, and each method trades memory, speed and accuracy differently. Quantization is one of the main levers for fitting an [open-weight model](/learn/open-weight-models) onto the hardware you have.

## What it saves

Models are usually loaded in bf16, which takes 2 bytes per parameter. Hugging Face's [overview](https://huggingface.co/docs/transformers/main/en/quantization/overview) works through an 8B model: about 16GB for its weights in bf16, about 8GB at 8 bits and about 4 to 5GB at 4 bits.

Those figures cover weights only. At long context or high batch sizes the [KV cache](/learn/kv-cache) can take a large share of memory too, which is why it can be quantized separately (see below).

## How the mapping works

A quantized number can take only a few values: 256 for int8 and 16 for int4. The Hugging Face [concept guide](https://huggingface.co/docs/transformers/main/en/quantization/concept_guide) explains that every method therefore introduces rounding error, and methods differ in how they keep it from hurting the output.

The common mapping is affine. It stretches the integer range over a tensor's minimum and maximum using a scale and a zero point, and dequantizing gives back an approximation of the original value. Symmetric quantization fixes the zero point at 0, which is cheaper but can leave some values unused.

How many values share one scale matters. The guide lists three choices:

| Granularity | Scales | Trade-off |
|---|---|---|
| Per-tensor | One for the whole tensor | Cheap, but one large outlier stretches the range for every other weight |
| Per-channel | One per output channel (a row of the weight matrix) | Narrower range per scale |
| Per-group | One per group of consecutive values, such as 128 weights | Best range fit, with extra storage for each scale |

Most low-bit methods quantize per group, and outliers are why. Dettmers and colleagues [found](https://arxiv.org/abs/2208.07339) that large models develop systematic outlier features that dominate attention. Their LLM.int8() method runs those dimensions in 16-bit and keeps over 99.9% of values in 8-bit.

## Weights, activations and the KV cache

| What is quantized | Examples | When it helps |
|---|---|---|
| Weights only | bitsandbytes 4-bit, AWQ, GPTQ, HQQ | Memory-bound decoding. The kernel dequantizes weights before each matmul. |
| Weights and activations (W8A8) | FP8 and INT8 schemes | Compute-bound work such as prefill and large batches. Needs hardware support for the low-precision matmul. |
| KV cache | Hugging Face `QuantizedCache` | Long contexts where the cache dominates memory |

The concept guide [explains](https://huggingface.co/docs/transformers/main/en/quantization/concept_guide) why weight-only methods do not always run faster. Generating one token at a time, a GPU spends most of each step reading weights from memory, so 4-bit weights move a quarter of the data bf16 weights do. That can outweigh the cost of dequantizing.

For the cache, Hugging Face's [docs](https://huggingface.co/docs/transformers/main/en/kv_cache) describe a quantized cache with hqq, quanto and fp8 backends. They warn it can harm latency when the context is short and GPU memory is enough without it.

## Data types

| Type | Notes | Source |
|---|---|---|
| int8, int4 | Evenly spaced integers; int4 has only 16 values, so it loses more accuracy | [Concept guide](https://huggingface.co/docs/transformers/main/en/quantization/concept_guide) |
| FP8 (E4M3, E5M2) | Floating point in 8 bits; E4M3 is the usual inference choice; matmuls need NVIDIA compute capability 8.9 or higher, AMD MI300 or Intel XPU | [Concept guide](https://huggingface.co/docs/transformers/main/en/quantization/concept_guide) |
| NF4 | 4-bit type with more values near zero, where most weights are | [Concept guide](https://huggingface.co/docs/transformers/main/en/quantization/concept_guide); QLoRA [paper](https://arxiv.org/abs/2305.14314) |
| MXFP4, NVFP4 | 4-bit floats with a scale per small block: 32 values and a power-of-two scale for MXFP4, 16 values and an FP8 scale (plus one FP32 per-tensor scale) for NVFP4 | [Concept guide](https://huggingface.co/docs/transformers/main/en/quantization/concept_guide) |
| Codebooks (AQLM, VPTQ) | Groups of weights stored as an index into learned vectors; used below 4 bits | [Concept guide](https://huggingface.co/docs/transformers/main/en/quantization/concept_guide) |

## When quantization happens

Post-training quantization (PTQ) works on a model that is already trained. Some PTQ methods need no data: the scales come from the weights, so quantization is fast enough to run as the model loads, as with bitsandbytes, HQQ and torchao. Others calibrate on a small dataset, which usually gives better accuracy at 4 bits and below but takes minutes to hours.

Two calibrated methods are widely used:

- **GPTQ.** Frantar and colleagues [describe](https://arxiv.org/abs/2210.17323) a one-shot method that uses approximate second-order information. It quantized a 175B-parameter model in about four GPU hours, to 3 or 4 bits per weight with negligible accuracy loss, and reported end-to-end speedups of about 3.25x on an A100 and 4.5x on an A6000 over FP16.
- **AWQ.** Lin and colleagues [report](https://arxiv.org/abs/2306.00978) that protecting only 1% of salient weights can greatly reduce quantization error, with those channels found from the activation distribution and no backpropagation or reconstruction.

Quantization-aware training (QAT) simulates the rounding during training so the model learns to tolerate it. The concept guide says it usually gives the best accuracy at very low bit widths but requires training the model. BitNet's 1.58-bit weights are trained this way.

## Choosing a method

Hugging Face's [method guide](https://huggingface.co/docs/transformers/main/en/quantization/selecting) recommends a starting point per goal. The table below follows it as of November 2026.

| Goal | Start with |
|---|---|
| Fit a model in less memory with the least setup | bitsandbytes (4 or 8 bits, no calibration) |
| Fine-tune with QLoRA | bitsandbytes with NF4 |
| Speed up inference with `torch.compile` | torchao |
| Best 4-bit accuracy | GPTQ, or AWQ and AutoRound quantized with their own libraries |
| FP8 on Ada, Hopper or Intel XPU | Fine-grained FP8 |
| FP4 on Blackwell | NVFP4 |
| Below 4 bits | HQQ or SINQ, or AQLM, VPTQ and SpQR checkpoints |

If a checkpoint is already quantized, the guide says the checkpoint decides the method. Most GGUF models, the [overview](https://huggingface.co/docs/transformers/main/en/quantization/overview) adds, are dequantized to full precision as they load in Transformers, so they use as much memory as an unquantized model there.

## How much accuracy you lose

Hugging Face's [benchmarks](https://huggingface.co/docs/transformers/main/en/quantization/selecting) on Llama 3.1 8B and 70B group the results by bit width:

| Bits | Memory vs bf16 | Accuracy |
|---|---|---|
| 8 | About 2x smaller | Very close to bf16 |
| 4 | About 4x smaller | High; AWQ and GPTQ often lead but need calibration |
| Below 4 | More than 4x smaller | Noticeable drop, especially at 2 bits |

An independent study agrees on the shape. Kurtic and colleagues [ran](https://arxiv.org/abs/2411.02355) over 500,000 evaluations on Llama-3.1 and found FP8 W8A8 "effectively lossless across all model scales," well-tuned INT8 W8A8 at 1 to 3% degradation, and INT4 weight-only "more competitive than expected, rivaling 8-bit quantization." They recommend W4A16 for cost efficiency in synchronous serving and W8A8 for asynchronous continuous batching. Both studies cover Llama models, so check your own.

## Test the quantized model on your task

Benchmark averages hide task-level drops. Hugging Face's guide tells readers to benchmark accuracy and speed on their own task and hardware before choosing a method for production. In practice that means running the same [evaluation](/learn/llm-evaluation) set against the full-precision and quantized versions, and keeping it as a [regression test](/learn/llm-regression-testing) when you swap checkpoints.

HoneyHive's [tracing concepts](https://docs.honeyhive.ai/v2/tracing/concepts.md) describe events that store model inputs, outputs and metadata such as token counts and cost, so you can compare latency, cost and outputs per call between two deployments. For the serving side, see [LLM inference](/learn/llm-inference) and [LLM cost optimization](/learn/llm-cost-optimization).

## FAQ

### What is quantization in an LLM?

It stores a model's weights, and sometimes its activations, in fewer bits so the model needs less memory. Hugging Face's [docs](https://huggingface.co/docs/transformers/main/en/quantization/overview) note that fewer bits cost some accuracy, and each method makes a different trade-off between memory, speed and accuracy.

### How much memory does quantization save?

Hugging Face's [overview](https://huggingface.co/docs/transformers/main/en/quantization/overview) gives an 8B model as about 16GB of weights in bf16, about 8GB at 8 bits and about 4 to 5GB at 4 bits. That counts weights only, not the KV cache.

### Does quantization hurt accuracy?

A little at 8 bits and 4 bits, more below that. Kurtic and colleagues [evaluated](https://arxiv.org/abs/2411.02355) Llama-3.1 models and found FP8 effectively lossless, well-tuned INT8 at 1 to 3% degradation, and 4-bit weight-only competitive with 8-bit. Hugging Face's [benchmarks](https://huggingface.co/docs/transformers/main/en/quantization/selecting) show a noticeable drop below 4 bits.

### What is the difference between GPTQ and AWQ?

Both are post-training, weight-only methods that calibrate on data. The Hugging Face [guide](https://huggingface.co/docs/transformers/main/en/quantization/concept_guide) says GPTQ adjusts the remaining columns to make up for each column's rounding error, while AWQ scales up the channels with the largest activations before rounding.

### Is quantization the same as fine-tuning?

No. Post-training quantization changes how weights are stored, not what the model learned. QLoRA combines the two: the [paper](https://arxiv.org/abs/2305.14314) freezes a 4-bit base model and trains small LoRA adapters on top.
