Terminal-Bench

Terminal-Bench, created by the Laude Institute with Stanford University, tests AI agents on real-world terminal tasks — the multi-step operational work developers and sysadmins actually do — inside sandboxed, reproducible containerized environments. Version 2.0 hardened the suite to 89 manually verified tasks (each receiving ~3 reviewer-hours), moved to the new Harbor evaluation framework, and eliminated tasks that were unsolvable for artificial reasons, becoming the benchmark virtually every frontier AI lab reports against.

The Stanford HAI 2026 AI Index made it famous beyond the lab: agent success on Terminal-Bench jumped from 20% to 77.3% in a single year, the headline number for the industry's claim that agents crossed a production-reliability threshold. A Terminal-Bench-Science spin-off (August 2026, 70 tasks across chemistry, biology, physics, and math) showed the frontier is jagged — even Claude Opus 5 manages only ~30% on real scientific workflows. Honest caveats: benchmark scores compress as tasks saturate, terminal tasks still underrepresent GUI and long-horizon planning, and vendor-reported scores should be checked against the independent leaderboard.

Clone the Terminal-Bench repo and run tasks via the Harbor framework in Docker.