Nanotron
Minimalistic Hugging Face library for 3D-parallelism LLM pretraining.
About
Nanotron is Hugging Face's lightweight library for pretraining transformer models from scratch, built around 3D parallelism: tensor, pipeline, and data parallelism, plus MoE, expert parallelism, and ZeRO-1. It keeps the codebase deliberately small — roughly twenty core source files — while using modern techniques like asynchronous tensor parallelism and 1F1B pipeline scheduling.
It trained the SmolLM family and Cosmo-1B, and Hugging Face publishes an Ultrascale Playbook documenting efficient scaling practices with it. Nanotron is a pretraining framework rather than a fine-tuning toolkit, so teams looking for post-training should reach for TRL, LLaMA-Factory, or Unsloth instead.
What it can do
- 3D parallelism (tensor, pipeline, data)
- MoE and expert parallelism support
- ZeRO-1 optimizer sharding
- Minimalist, hackable codebase
- YAML-driven training configuration
Setup
pip install nanotron, then run training with run_train.py and YAML configs; Slurm launcher included for clusters.
Prerequisites
Multi-GPU or multi-node environment, PyTorch with CUDA, and a tokenized dataset.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 79; Trending score 80/100 from researcher assessment.
★ 2,800 GitHub stars
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.