Llemma-7B

Llemma is an open language model for mathematics from EleutherAI, released October 16, 2023 (arXiv:2310.10631). The recipe continues pretraining Code Llama on Proof-Pile-2, a 55-billion-token mixture of mathematical text and code, plus the 11B-token AlgebraicStack dataset of math-related code. The resulting 7B and 34B base models were the new state of the art for publicly released base models on the MATH benchmark, and the authors demonstrated tool use — Python interpreters and formal theorem provers — as well as open-sourcing all training data and code, a contrast with closed math models like Minerva.

Limitations: Llemma models are base models, not instruction-tuned chat models, so they need prompting or fine-tuning for assistant behavior. They inherit Code Llama's lineage, so verify the exact license terms on the Hugging Face model card before commercial use rather than assuming a permissive license. As a 2023-era model it has been superseded on math benchmarks by newer specialists such as DeepSeekMath, Qwen2.5-Math, and AceMath, but it remains a clean, fully open research platform.