OpenELM-3B

OpenELM-3B is the largest variant of OpenELM (Open-source Efficient Language Models), released by Apple in April 2024. It uses a layer-wise scaling strategy where attention heads and feed-forward dimensions grow deeper in the network, improving accuracy without increasing parameter count. Apple released the full training recipe: weights, CoreNet training code, logs, and the 1.8T-token pretraining data mixture (RefinedWeb, deduplicated PILE, RedPajama and Dolma subsets).

The 3B model ships in pretrained and instruction-tuned forms with a 2,048-token context window and MLX support for Apple Silicon. Honest caveat: the weights are public under the Apple Sample Code License, which is not an OSI-approved open license, so commercial-use terms need review. It remains a landmark for reproducible small-model research.