PhoGPT-7B5

PhoGPT-7B5 is a 7.5-billion-parameter Vietnamese language model from VinAI (Vingroup), unveiled at AI Day 2023 on December 5. Unlike adapted multilingual models, it was pretrained from scratch on a 41GB Vietnamese corpus with a Vietnamese-specific byte-level BPE tokenizer, using flash attention and ALiBi for context extrapolation. An instruction-tuned variant, PhoGPT-7B5-Instruct, was fine-tuned on 150K Vietnamese prompt-response pairs.

VinAI reported performance near GPT-3.5-turbo on Vietnamese evaluations and released it open-source for commercial use. The team later recommended the smaller PhoGPT-4B series for better efficiency. Technical report: arXiv 2311.02945.