AudioGen

AudioGen, from Meta's FAIR team (arXiv September 2022, ICLR 2023), tackles textually guided audio generation: generating environmental sounds and effects from text captions. It is an autoregressive transformer that operates on discrete audio tokens from a neural audio codec, conditioned on text encoded by a pretrained T5 model, which lets it generalize to sound concepts absent from the limited text-audio training data.

The team curated ten datasets to handle text-audio scarcity and used mix-based augmentation to teach the model to separate overlapping sources. Meta reimplemented it in the open AudioCraft library as a single-stage autoregressive transformer over a 16kHz EnCodec tokenizer, so usable open weights exist even though the published checkpoints differ from the paper's. A pioneer of the text-to-audio category alongside later systems like Stable Audio.