TorchTitan

TorchTitan is Meta's PyTorch-native training platform for generative AI, and it is essentially the open-sourced version of how Meta trains large models. You get ready-made, configurable recipes for Llama 3 and 4, Qwen3, and DeepSeek-V3 style architectures, with the distributed-training machinery that makes large runs work: FSDP2, tensor/pipeline/context parallelism, torch.compile integration, Float8 training, and checkpointing that survives the realities of multi-hundred-GPU jobs.

The design philosophy is composability over magic: everything is plain PyTorch 2.x components you can read and modify, rather than a framework that hides the training loop from you. At 5,800+ stars and BSD-3 it is the reference starting point for anyone training LLMs on PyTorch at scale. The honest caveat is the audience: this is for teams with serious GPU budgets and distributed-training experience. If you are fine-tuning a 7B model on one machine, Torchtune or Axolotl will waste far less of your week.

1. Install a recent PyTorch (nightly recommended per README) and clone the repo.
2. Install torchtitan from source: pip install -e . in the repo.
3. Pick a recipe config (e.g. Llama 3 8B debug config) and adjust for your model and hardware.
4. Launch distributed training with torchrun using the documented parallelism flags (FSDP2, tensor/pipeline parallelism, torch.compile).
5. Monitor with the built-in TensorBoard/weight metrics and use async checkpointing for long runs.