lm-evaluation-harness
The standard harness for benchmarking language models.
About
lm-evaluation-harness is EleutherAI's framework for evaluating language models: 60+ standard benchmarks (MMLU, ARC, HellaSwag, TruthfulQA, GSM8K...) with reproducible, versioned task definitions — the engine behind Hugging Face's Open LLM Leaderboard. 14.2k stars, MIT. Run any HF model or API backend through the same tasks and compare apples to apples. Caveat: public-benchmark contamination is a real debate, so treat scores as directional; and large-model evals are slow and expensive — the harness standardizes the process, it doesn't make it cheap.
What it can do
- **60+ benchmarks**: MMLU, ARC, HellaSwag, TruthfulQA, GSM8K, and more
- **Reproducible**: versioned tasks, fixed prompts and metrics
- **Multi-backend**: HuggingFace, vLLM, OpenAI, Anthropic, and others
- **Leaderboard engine**: powers the Open LLM Leaderboard
- **Custom tasks**: YAML-defined tasks for your own evals
Setup
```bash
pip install lm-eval
lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-7B-Instruct --tasks mmlu,arc_challenge --limit 50
```
Docs: github.com/EleutherAI/lm-evaluation-harness.
Prerequisites
Python 3.9+. GPU for local models, or API keys for hosted backends.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: 14,197 GitHub stars (log-scaled vs catalog peers); last push 28d ago; 5 releases/12mo; 1165 open issues. Manual review: standard LLM benchmark harness (60+ tasks), powers Open LLM Leaderboard; 14.2k stars; contamination debates make scores directional; big-model evals slow/expensive. License: MIT.
★ 14,197 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 7.8/10
- Maintenance · recency, releases, issue load
- 6.5/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · 14,197 GitHub stars (log-scaled vs catalog peers)
- · last push 28d ago; 5 releases/12mo; 1165 open issues
- · manual review: standard LLM benchmark harness (60+ tasks), powers Open LLM Leaderboard; 14.2k stars; contamination debates make scores directional; big-model evals slow/expensive
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.