M2D2

M2D2 is a second-generation masked modeling duo for audio proposed by Daisuke Niizumi and colleagues at NTT, published on arXiv in March 2025. It addresses a gap in contrastive language-audio pretraining (CLAP): while CLAP aligns audio and text well, its audio features generalize poorly to pure audio tasks, whereas self-supervised models learn strong general audio features but lack language grounding.

M2D2 combines an SSL masked-modeling duo (M2D) with CLAP in a two-stage pretraining process, using LLM-based sentence embeddings for semantic supervision. The first stage learns generalizable audio features aligned with fine-grained semantic embeddings; the second stage learns CLAP features from those audio features.

The paper reports state-of-the-art fine-tuning mAP of 49.0 on AudioSet, state-of-the-art performance on music tasks, and top-level results on audio-language benchmarks. It is a research model aimed at audio representation learning rather than a consumer product.