SALMONN

SALMONN (Speech Audio Language Music Open Neural Network) is a collaboration between Tsinghua's Department of Electronic Engineering and ByteDance that gives a large language model generic hearing. It fuses a Whisper speech encoder and a BEATs general-audio encoder through a window-level Q-Former, then aligns the result into a Vicuna 7B or 13B backbone with a LoRA adaptor. One model can transcribe and translate multilingual speech, caption audio events, describe music, do audio-speech co-reasoning, and hold open-ended dialogue about what it hears, trained on 600k+ QA samples. The code and weights are Apache 2.0, which is unusually permissive for this class of model. It is an understanding model, not a generator — it will not synthesize audio — and the 13B variant wants around 26 GB of VRAM.

Team-reported: strong results across 8 instruction tasks plus emergent cross-modal tasks unseen in training (paper arXiv 2310.13289, ICLR 2024)