Fugaku-LLM

Fugaku-LLM was released on May 10, 2024, by a Japanese research consortium led by Tokyo Institute of Technology, RIKEN, Fujitsu, Tohoku University, Nagoya University, CyberAgent, and Kotoba Technologies. It is a 13-billion-parameter model trained on 380 billion tokens using 13,824 nodes of the Fugaku supercomputer, notable because Fugaku runs on Arm-based CPUs rather than GPUs, requiring the team to port Megatron-DeepSpeed to the machine.

Trained from scratch on proprietary Japanese data from CyberAgent plus English data, it scored 5.5 on Japanese MT-Bench, the highest among open models trained on Japan-produced data, with 9.18 on humanities and social sciences tasks. Source code is on GitHub and weights on Hugging Face, usable for research and commercial purposes under the project's license.