Hunyuan-Large

Hunyuan-Large is Tencent's flagship open mixture-of-experts language model, released November 2024 with 389 billion total parameters and 52 billion activated per token. At launch it was the largest open-source Transformer-based MoE model, with a 256K token context window. Tencent reports it outperforms Llama 3.1 70B and is competitive with Llama 3.1 405B on language, reasoning, math, and coding benchmarks, with 88.4 percent on MMLU and 92.8 percent on GSM8K cited from the technical report.

The paper highlights large-scale synthetic data, mixed expert routing, KV-cache compression, and expert-specific learning rates as key techniques. Code and checkpoints were released publicly; note the release uses Tencent's Hunyuan community license terms rather than a standard OSI license, so check the model card before commercial use.