EvalPlus
Rigorous code-synthesis evaluation framework with augmented HumanEval+ and MBPP+.
About
EvalPlus is a code synthesis evaluation framework created by researchers at UIUC and published at NeurIPS 2023. It addresses test inadequacy in LLM coding benchmarks by extending HumanEval with 80 times more test cases per problem through LLM-based and mutation-based automatic test generation.
HumanEval+ reduces reported pass at 1 scores by 19 to 29 percent compared to original HumanEval, exposing mis-rankings where models like WizardCoder and Phind-CodeLlama outperform ChatGPT. The framework also ships MBPP+ and EvalPerf for efficiency evaluation, runs in sandboxed Docker containers across OpenAI, Anthropic, Gemini, vLLM, and HuggingFace backends, and is open source under Apache 2.0. It is widely adopted by Meta Llama, Qwen, DeepSeek-Coder, StarCoder2, and Snowflake Arctic teams.
What it can do
- HumanEval+ with 80x more test cases per problem
- MBPP+ sanitized and augmented suite
- EvalPerf for code efficiency measurement
- Sandboxed Docker execution
- Multi-backend support including vLLM
- Public leaderboard and open datasets
Setup
pip install evalplus and run the HumanEval+ or MBPP+ evaluation scripts against your model.
Prerequisites
Python environment; model API access or local weights for the backend under test.
Use cases
Works with
Trust score — 7/10
Scored 7/10 (metrics tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 91; Trending score 68/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.