Skip to content
← Models
Open weightsGoogle/10 trustVerified

MetricX-24

Google's top-scoring neural MT metric at WMT24, built on mT5 and trained on a decade of human quality judgments.

Visit official page →

About

MetricX-24 is Google's submission to the WMT24 metrics shared task, where it ranked among the top systems. It is a regression model initialized from mT5 and fine-tuned on direct-assessment and MQM human judgments from WMT 2015 through 2022, plus synthetic examples designed to reduce bias against longer translations. Released checkpoints span Large, XL, and XXL, with Hybrid variants that also work in reference-free quality estimation mode.

Unlike Unbabel's metric family, MetricX-24 is Apache-2.0 licensed, which makes it the practical choice for commercial MT evaluation pipelines. Scores are clipped to a 0-25 range. Google recommends XXL for the best agreement with human judgments, Large for speed, and XL as the middle ground.

Specs

Context window

1024 tokens (mT5 encoder)

Pricing

Free (open weights, Apache-2.0)

Free quota

Free — self-hosted weights

API access

No official API; Hugging Face weights

Weights

Open weights

Released

Not disclosed

Benchmarks

Author-reported (WMT24 metrics shared task): top-ranking system by correlation with human judgments

Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.

Use cases

mt-evaluationquality-estimationsystem-rankingtranslation-qa

Trust score — 7/10

/10 trustVerified

Scored 7/10 (verified tier, metrics-v1). Evidence: Google release with WMT24 metrics-shared-task standing confirmed via the Hugging Face model cards; Large/XL/XXL Hybrid checkpoints published. Strengths: top WMT24 correlation with human judgments, permissive Apache-2.0 license, hybrid reference-free mode. Docked for: (1) newer than COMET-family metrics with a smaller independent-evaluation track record; (2) XXL size is heavy for routine use. License: Apache-2.0.

How the trust score is built

Full methodology →
Adoption · stars/downloads, log-scaled vs peers
7.0/10
Maintenance · recency, releases, issue load
7.0/10
Openness · license + docs
10.0/10
Manual review · sanity read + red-flag check
7.0/10
  • · Google release; top system at the WMT24 metrics shared task
  • · Large/XL/XXL Hybrid checkpoints, mT5-initialized, trained on WMT15-22 DA+MQM
  • · Scores clipped to 0-25 range

Sources