StableLM 2 1.6B

StableLM 2 1.6B was released by Stability AI in January 2024 as the first model in its Stable LM 2 family of multilingual small language models. Pretrained on 2 trillion tokens of filtered open data (supplemented with multilingual CulturaX data) on 512 NVIDIA A100 GPUs, it handles English, Spanish, German, Italian, French, Portuguese, and Dutch.

At release it set state-of-the-art results for models under 2B parameters on the Open LLM Leaderboard, outscoring Phi-1.5, TinyLlama 1.1B, and Falcon 1B. Both base and instruction-tuned checkpoints were published, with unusually transparent training details including the final pre-training checkpoint and optimizer states. Date is era-level (January 2024).