Chinchilla

Chinchilla is a 70-billion-parameter language model from DeepMind that exists mainly to prove a point about how language models should be trained. The March 2022 paper 'Training Compute-Optimal Large Language Models' by Hoffmann et al. re-examined OpenAI's Kaplan scaling laws with data size treated as a first-class variable, fitting a power-law surface across more than 400 training runs. The finding: for a fixed compute budget, model size and training tokens should scale in equal proportion, roughly 20 tokens per parameter.

To validate this, DeepMind trained Chinchilla on 1.4 trillion tokens using the same compute budget as its 280B-parameter Gopher, which had seen only 300 billion tokens. Despite being four times smaller, Chinchilla outperformed Gopher on nearly every benchmark. The result implied that flagship models of the era, including GPT-3 at 175B parameters on 300B tokens, were massively under-trained, and it reshaped how every major lab budgets pretraining compute. Chinchilla itself was never released to the public: it is a research artifact, so openWeights is false. Its influence is the scaling rule it established.