Pangu Ultra

Pangu Ultra is Huawei's dense large language model with 135 billion parameters, described in the paper arXiv:2504.07866 published April 10, 2025. It is the flagship of the Pangu line's push to train frontier-scale models exclusively on Chinese hardware: the full training run used 8,192 Ascend NPUs on 13.2 trillion tokens, with context extended from 4K to 128K.

The technical contributions include depth-scaled sandwich normalization and tiny initialization to eliminate loss spikes in very deep models. Per the paper's evaluations, Pangu Ultra advances the state of the art for dense LLMs against models like Llama 405B and Mistral Large 2, and is competitive with sparse models like DeepSeek-R1. Weights are not openly released; the model is served to commercial customers through Huawei Cloud.