# How Should AI Safety Benchmarks Benchmark Safety?

arXiv: https://arxiv.org/abs/2601.23112 · HTML: https://mlsanity.com/paper/2601.23112/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2026.

## Built on by (most cited first, 5 of 5)

- [Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents](https://mlsanity.com/paper/2605.16282/) (2026)
- [Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM Agents](https://mlsanity.com/paper/2602.16943/) (2026)
- [Making AI Evaluation Deployment Relevant Through Context Specification](https://mlsanity.com/paper/2603.06811/) (2026)
- [Studying People to Study AI: Expert Perspectives on the Epistemic Fit and Barriers of Human Research in AI Safety & Ethics](https://mlsanity.com/paper/2608.05656/) (2026)
- [Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark](https://mlsanity.com/paper/2610.12409/) (2026)

## Abstract (arXiv)

AI safety benchmarks are pivotal for safety in advanced AI systems; however, they have significant technical, epistemic, and sociotechnical shortcomings. We present a review of 210 safety benchmarks that maps out common challenges in safety benchmarking, documenting failures and limitations by drawing from engineering sciences and long-established theories of risk and safety. We argue that adhering to established risk management principles, mapping the space of what can(not) be measured, developing robust probabilistic metrics, and efficiently deploying measurement theory to connect benchmarking objectives with the world can significantly improve the validity and usefulness of AI safety benchmarks. The review provides a roadmap on how to improve AI safety benchmarking, and we illustrate the effectiveness of these recommendations through quantitative and qualitative evaluation. We also provide workflow-oriented guiding questions with illustrative benchmark that help researchers and practitioners develop robust and epistemologically sound safety benchmarks. This study advances the science of benchmarking and helps practitioners deploy AI systems more responsibly.