WalledEval
Safety evaluation toolkit probing LLMs with 35+ adversarial benchmarks.
About
WalledEval is an open-source Python toolkit from Walled AI Labs for comprehensive safety evaluation of large language models, introduced in an August 2024 arXiv paper. It bundles more than 35 safety benchmarks covering jailbreaks, prompt injections, multilingual safety, and culturally contextualized harms, and it ships with custom mutators that rewrite prompts into adversarial variants to probe model weaknesses more deeply than static test sets.\n\nIt supports both open-weight models through Hugging Face Transformers and API-based models from major providers, and it also introduces WalledGuard, a content moderation tool, alongside specialized datasets. It is installed from PyPI as walledeval and aimed at researchers and red teams who need a one-stop safety harness rather than hand-rolling evaluations per model.
What it can do
- 35+ safety benchmarks covering jailbreaks, injections, and multilingual safety
- Custom text mutators generating adversarial prompt variants
- Support for open-weight HF models and major API providers
- WalledGuard content moderation tool and specialized safety datasets
- Paper-backed methodology with reproducible evaluation pipelines
Setup
Run pip install walledeval and load models via the HF_LLM wrapper or API provider integrations.
Prerequisites
Python with PyTorch, plus a model checkpoint or provider API keys.
Use cases
Works with
Trust score — 7/10
Scored 7/10 (metrics tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 100; Trending score 70/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.