AudioLM

AudioLM, published by Google Research in September 2022, is the landmark paper that made audio generation a language-modeling problem. It maps raw audio into discrete tokens via a hybrid scheme — semantic tokens from discretized w2v-BERT activations capture long-term structure like speech content and musical motifs, while acoustic tokens from a SoundStream neural codec capture timbre, speaker identity, and recording conditions — then trains a Transformer on the token sequence.

Trained only on raw waveforms with no transcripts or annotations, it learned to generate syntactically plausible speech continuations that preserved an unseen speaker's identity and prosody from a short prompt, plus coherent piano music continuations despite no symbolic music representation. It directly shaped SoundStorm, MusicLM, VALL-E, and Voicebox.

Google never officially released AudioLM weights or code — it is a research framework, not a downloadable model. Community re-implementations (e.g. lucidrains' PyTorch version) exist unofficially.