mT5

mT5 is the multilingual variant of Google's Text-to-Text Transfer Transformer, released in October 2020. It follows the T5 recipe closely but is pretrained on mC4, a multilingual Common Crawl corpus covering 101 languages, in sizes from 300 million to 13 billion parameters.

At launch it set state-of-the-art results on multilingual benchmarks and became a workhorse for cross-lingual transfer, especially for lower-resource languages. It is released under Apache 2.0 and all checkpoints are available on Hugging Face.