Tau-bench

Tau-bench (styled τ-bench) is a benchmark from Sierra Research that tests whether an AI agent can actually do a customer-service job, not just answer questions about one. The agent talks to a simulated user over multiple turns while calling backend tools — checking orders, modifying bookings, applying refund policies — and it is graded on whether the task ends in the right state. That combination of dialogue, tool use, and policy-following is what makes it one of the more credible public signals of production-grade agent quality.

The original paper (Yao et al., arXiv 2406.12045, June 2024) covered retail and airline domains with 165 tasks. The current version is τ²-bench (arXiv 2506.07982, June 2025), which adds telecom and banking-knowledge domains and a proper train/test split; the original tau-bench repo is deprecated in its favor. Scoring uses task success with pass^k reliability estimates, and there is a public leaderboard at taubench.com. Both repos are MIT licensed.

1. Clone https://github.com/sierra-research/tau2-bench and install with pip.
2. Set API keys for the agent model, the user simulator, and the judge.
3. Run the benchmark CLI against a domain (airline, retail, telecom).
4. Compare task-success and pass^k scores against the public leaderboard.