Inspect
The UK AI Security Institute's framework for LLM capability and safety evals.
About
Inspect is the open-source evaluation framework built by the UK AI Security Institute: run structured evals of model capabilities, safety risks, and agentic workflows, with 200+ pre-built evaluations ready to run on any model. It handles prompt engineering, tool use (executed in Docker/local sandboxes), multi-turn dialog, and model-graded scoring, recording everything to a log store with a web viewer. Extensions ship as Python packages. Docs at inspect.aisi.org.uk ship llms.txt files so coding agents can consume them directly. MIT, 2.9M PyPI downloads/month.
What it can do
- **200+ pre-built evals**: capabilities, safety, and agentic benchmarks ready to run
- **Sandboxed tool use**: model-generated bash/python runs contained in Docker/local sandboxes
- **Model-graded evals**: LLM judges plus deterministic scorers
- **Log store + viewer**: full transcripts of every eval run
- **Extensible**: new tasks/scorers as Python packages
- **Agent-readable docs**: llms.txt / llms-full.txt published for coding agents
Setup
```bash
pip install inspect-ai
```
```bash
# Run a built-in eval, e.g. against a local model
inspect eval inspect_evals/mmlu --model vllm/meta-llama/Llama-3.2-1B
inspect view # log viewer
```
Model-generated tool calls run sandboxed (Docker/local). Docs: inspect.aisi.org.uk.
Prerequisites
Python 3.10+. Model access: API keys or local serving (vLLM, Ollama, HF Transformers). Docker for sandboxed tool-use evals.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: 2,990 GitHub stars (log-scaled vs catalog peers); blended with ~2,896,739 monthly package downloads; last push 0d ago; 1 releases/12mo; 402 open issues. Manual review: UK AI Security Institute eval framework; 200+ pre-built evals, sandboxed tool use; 2.9M PyPI/mo; narrower community (3k stars); safety-first roadmap. License: MIT.
★ 2,990 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 7.5/10
- Maintenance · recency, releases, issue load
- 6.4/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · 2,990 GitHub stars (log-scaled vs catalog peers)
- · blended with ~2,896,739 monthly package downloads
- · last push 0d ago; 1 releases/12mo; 402 open issues
- · manual review: UK AI Security Institute eval framework; 200+ pre-built evals, sandboxed tool use; 2.9M PyPI/mo; narrower community (3k stars); safety-first roadmap
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.