TorchAO
PyTorch-native quantization and sparsity library that composes with torch.compile.
About
TorchAO (PyTorch Architecture Optimization) is PyTorch's official library for model optimization, maintained under the pytorch/ao repository. It provides quantization (INT8, INT4, FP8) and 2:4 sparsity workflows that compose with torch.compile, designed for both training and inference optimization. Released under BSD-3 in September 2024, it supersedes earlier one-off quantization approaches and integrates with the broader PyTorch 2.x optimization stack.
The honest caveats: quantization quality is model- and task-dependent — aggressive INT4 quantization can visibly degrade outputs on some workloads, so calibration and eval are required. API surface is still evolving as the library matures, and getting peak speedups requires correct hardware (e.g. FP8 on H100-class GPUs). It complements rather than replaces libraries like Optimum and AutoAWQ; pick TorchAO when your stack is PyTorch-native end to end.
What it can do
- INT8/INT4/FP8 quantization
- 2:4 structured sparsity
- Composable with torch.compile
- Training and inference optimization
- Lower-precision model serialization
Setup
pip install torchao; apply quantization workflows to your PyTorch model.
Prerequisites
PyTorch 2.x environment; ideally torch.compile-compatible setup.
Use cases
Works with
Trust score — 9/10
Scored 9/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 85; Trending score 86/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.