MMAudio

MMAudio generates high-quality audio synchronized to video, with optional text conditioning. Its trick is multimodal joint training: instead of training only on limited video-audio pairs, it trains jointly on large-scale text-audio data too, which teaches it what good audio sounds like in general. A conditional synchronization module then aligns the video frames with the audio latents, so effects land on the action instead of drifting.

The December 2024 paper (accepted at CVPR 2025) reports state of the art among public models on audio quality, semantic alignment and audio-visual sync, and the numbers are striking for the size: the 157M model generates an 8-second clip in about 1.23 seconds. Three sizes (157M, 621M, 1.03B) and two sample rates (16kHz, 44.1kHz) were trained, and code, checkpoints and a demo are public. A v2 checkpoint later traded some benchmark scores for better real-world generalization. The exact license on the weights is not clearly stated, so check the repo before commercial use.

Provider-reported: ~10% lower Frechet Distance, ~15% higher Inception Score, ~4% higher ImageBind, ~14% better sync score vs baselines; no independent third-party benchmark found.