MERT
Self-supervised music understanding model: a HuBERT-style acoustic encoder with musical teachers, SOTA across 14 MIR tasks.
About
MERT is a self-supervised model for understanding music audio rather than generating it. It trains a BERT-style transformer with masked prediction, guided by two teacher models: an acoustic teacher based on a residual vector-quantized variational autoencoder, and a musical teacher based on the constant-Q transform, which captures the pitch structure that makes music different from speech. An in-batch noise mixup trick hardens the representations, and the design scales from 95M to 330M parameters.
The paper reports state-of-the-art overall scores across fourteen music understanding tasks, from tagging and genre classification to key detection, instrument and pitch classification, and beat tracking, and the model is evaluated under the MARBLE benchmark protocol. Published at ICLR 2024 with Apache-2.0 code and weights, it is the closest thing music has to a general-purpose audio foundation encoder, though its community is modest compared to speech counterparts like wav2vec 2.0.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
- Paper (arXiv 2306.00107): SOTA overall scores across 14 music understanding tasks (author-reported)
- ICLR 2024; evaluated under the MARBLE benchmark protocol
- No independently verified headline numbers
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 7/10
Scored 7/10 (TIER verified, metrics-v1). Evidence: ICLR 2024 paper (arXiv 2306.00107) with 14-task evaluation; Apache-2.0 official repo (yizhilll/MERT, ~517 stars); MARBLE benchmark evaluation. Strengths: peer-reviewed, permissive license, rare music foundation encoder. Docked for: (1) modest community (~517 stars); (2) headline claims are author-reported. License: Apache-2.0.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 6.0/10
- Maintenance · recency, releases, issue load
- 5.0/10
- Openness · license + docs
- 9.0/10
- Manual review · sanity read + red-flag check
- 7.0/10
- · ICLR 2024 paper (arXiv 2306.00107): RVQ-VAE acoustic teacher + CQT musical teacher, 95M-330M params
- · Official repo yizhilll/MERT, Apache-2.0, ~517 stars; SOTA on 14 music understanding tasks (author-reported)
- · Evaluated under the MARBLE benchmark protocol