DeepSpeed

DeepSpeed is Microsoft Research's deep-learning optimization library: ZeRO sharding lets you train models far larger than a single GPU's memory (optimizer states, gradients, and parameters partitioned across devices, with NVMe/CPU offload), and DeepSpeed-Inference brings tensor parallelism and kernel fusion to serving. 43.2k stars, Apache-2.0, 1M PyPI downloads/month. It's the library behind many of the largest open-model training runs. The price: a genuine learning curve — ZeRO stages, offload configs, and fused kernels reward careful tuning and punish casual defaults.

```bash

pip install deepspeed

```

```json

// ds_config.json — ZeRO-3 with offload

{"zero_optimization": {"stage": 3, "offload_optimizer": {"device": "cpu"}}}

```

```bash

deepspeed train.py --deepspeed ds_config.json

```

Docs: deepspeed.ai.