LM Evaluation Harness
The standard open-source harness for evaluating language models across 100+ benchmarks with one CLI.
About
EleutherAI's lm-evaluation-harness is the de facto standard for running LLM benchmarks reproducibly: over a hundred tasks (MMLU, HellaSwag, ARC, GSM8K, HumanEval, and many more) behind a single lm_eval command, with Hugging Face model integration out of the box.
It is what most open-model releases cite when they publish benchmark numbers, which makes it essential tooling for anyone comparing models seriously. The framework supports custom tasks and multiple backends (HF transformers, vLLM, API models), and it is MIT licensed with very active maintenance. If you only adopt one eval tool, this is the one.
What it can do
- 100+ built-in benchmark tasks with standardized metrics
- Single CLI (lm_eval) across HF, vLLM, and API-backed models
- Custom task authoring with decontamination tooling
- Reproducible configs matching published model leaderboards
- Caching and batched inference for large eval runs
Setup
pip install lm-eval, then run lm_eval --model hf --model_args pretrained=<model> --tasks mmlu,hellaswag --batch_size auto.
Prerequisites
Python 3.9+; a model to evaluate (local weights or API access).
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: 14,233 GitHub stars (EleutherAI/lm-evaluation-harness); pushed 2026-09-14 (recent activity); MIT license confirmed in repo; maintained by EleutherAI; officialUrl https://github.com/EleutherAI/lm-evaluation-harness curl-verified 200; PyPI lm-eval verified 200. Manual review: 100+ task coverage and lm_eval CLI workflow verified against repo docs; de-facto standard status supported by widespread model-release citations. License: MIT.
★ 14,233 GitHub stars
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.