Model quantization for deployment

The short version

Quantization shrinks model weights to fewer bits — fp16 to int8 to int4 — trading a little accuracy for a lot less memory and much faster inference. It's the reason a 70B model can run on a single workstation GPU and why your phone can do on-device inference at all. For deployment, it's not an optimization; it's the whole game.

What the levels mean

• fp16/bf16. The training default. Your baseline — everything is measured against this.
• int8. Roughly halves memory vs fp16 with quality loss that's usually within noise for large models. The safe first step.
• int4. Quarters memory. Modern methods (GPTQ, AWQ, GGUF quants) keep quality surprisingly intact on capable models, but small models degrade faster — a 7B model at int4 loses more, proportionally, than a 70B one.
• Below int4 (int2, ternary, 1-bit). Research frontier: works for specific architectures with careful training, not a drop-in choice yet.

What actually matters in practice

Memory bandwidth, not FLOPs. Inference is bottlenecked by moving weights from VRAM to the compute units, not by the math itself. Halving weight size nearly halves per-token latency — that's why quantization speedups are so dramatic even though the "compute" barely changed.

Bigger models quantize better. A 70B model at int4 typically beats a 13B model at fp16 on quality while using similar memory. If you're choosing between "small and precise" vs "big and quantized," bet on big and quantized.

Calibration data matters. The best methods use a small calibration set to decide how to round each layer. Mismatched calibration (code model calibrated on chat data) is a quiet quality killer.

KV cache is the other half. At long contexts, the KV cache — not the weights — dominates memory. Quantizing the cache (int8 KV) is what makes 100k+ context windows deployable.

Picking a format

GGUF dominates local/CPU inference (llama.cpp ecosystem), AWQ/GPTQ rule NVIDIA GPU serving, and each inference engine has its favorite. Choose the format your serving stack supports first, then the bit-width your quality evals allow.

Rule of thumb: quantize the biggest model that fits your memory budget at int4 before settling for a smaller model at higher precision. And always re-run your evals after quantizing — "usually fine" is not a measurement.