Pythia

Pythia is EleutherAI's 2023 suite of decoder-only language models purpose-built for scientific study: 8 sizes from 70M to 12B parameters, each trained on the Pile and on a deduplicated Pile in exactly the same data order, with 154 intermediate checkpoints per model — all publicly released. It remains one of the only suites satisfying all three: multiple scales, identical data order, and open checkpoints.

Designed for interpretability, scaling, and memorization research rather than raw capability, it became the standard testbed for studying how bias evolves during training, how data order affects memorization, and how term frequency interacts with model scale. It is released under Apache 2.0.

Practical note: Pythia models were renamed in January 2023 and an early v0 release was retrained, so verify exact checkpoint names on Hugging Face. Not competitive with modern instruction-tuned models for chat use.