GPTQ
One-shot 3-4 bit post-training quantization for generative transformers.
About
GPTQ is a landmark quantization method from IST Austria and ETH Zurich that compresses generative transformers to 3 or 4 bits per weight using approximate second-order (Hessian) information to minimize per-layer reconstruction error. It famously shrunk a 175B-parameter model into a single GPU in about four GPU hours with negligible accuracy loss.
Published at ICLR 2023, it was one of the first practical quantization methods for 100B+ models and remains a widely used baseline. It delivers multi-x end-to-end inference speedups on hardware like the NVIDIA A100 and A6000.
What it can do
- 3-bit and 4-bit post-training quantization
- Hessian-guided error compensation
- 175B models on a single GPU
- 2-bit and ternary regimes supported
Setup
Clone github.com/IST-DASLab/gptq and run the quantization script with a calibration dataset.
Prerequisites
Python, PyTorch, CUDA GPU, and a small calibration text set.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 76; Trending score 75/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.