# Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methods

arXiv: https://arxiv.org/abs/2505.05541 · HTML: https://mlsanity.com/paper/2505.05541/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2025.

## Built on by (most cited first, 15 of 15)

- [Quantifying CBRN Risk in Frontier Models](https://mlsanity.com/paper/2510.21133/) (2025)
- [Capabilities Ain't All You Need: Measuring Propensities in AI](https://mlsanity.com/paper/2602.18182/) (2026)
- [Embodied AI: Emerging Risks and Opportunities for Policy Action](https://mlsanity.com/paper/2509.00117/) (2025)
- [Measuring What AI Systems Might Do: Towards A Measurement Science in AI](https://mlsanity.com/paper/2603.00063/) (2026)
- [Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift](https://mlsanity.com/paper/2603.26676/) (2026)
- [Against racing to AGI: Cooperation, deterrence, and catastrophic risks](https://mlsanity.com/paper/2507.21839/) (2025)
- [The 2025 OpenAI Preparedness Framework does not guarantee any AI risk mitigation practices: a proof-of-concept for affordance analyses of AI safety policies](https://mlsanity.com/paper/2509.24394/) (2025)
- [Safe for Whom? Rethinking How We Evaluate the Safety of LLMs for Real Users](https://mlsanity.com/paper/2512.10687/) (2025)
- [A Review of Developmental Interpretability in Large Language Models](https://mlsanity.com/paper/2508.15841/) (2025)
- [CASE: An Agentic AI Framework for Enhancing Scam Intelligence in Digital Payments](https://mlsanity.com/paper/2508.19932/) (2025)
- [Physical Foundation Models: Fixed hardware implementations of large-scale neural networks](https://mlsanity.com/paper/2604.27911/) (2026)
- [SysAdmin: Measuring Instrumental Power-Seeking in Frontier AI](https://mlsanity.com/paper/2607.18239/) (2026)
- [Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking](https://mlsanity.com/paper/2507.08014/) (2025)
- [BlueGlass: A Framework for Composite AI Safety](https://mlsanity.com/paper/2507.10106/) (2025)
- ["Just a strange pic": Evaluating 'safety' in GenAI Image safety annotation tasks from diverse annotators' perspectives](https://mlsanity.com/paper/2507.16033/) (2025)

## Abstract (arXiv)

As frontier AI systems advance toward transformative capabilities, we need a parallel transformation in how we measure and evaluate these systems to ensure safety and inform governance. While benchmarks have been the primary method for estimating model capabilities, they often fail to establish true upper bounds or predict deployment behavior. This literature review consolidates the rapidly evolving field of AI safety evaluations, proposing a systematic taxonomy around three dimensions: what properties we measure, how we measure them, and how these measurements integrate into frameworks. We show how evaluations go beyond benchmarks by measuring what models can do when pushed to the limit (capabilities), the behavioral tendencies exhibited by default (propensities), and whether our safety measures remain effective even when faced with subversive adversarial AI (control). These properties are measured through behavioral techniques like scaffolding, red teaming and supervised fine-tuning, alongside internal techniques such as representation analysis and mechanistic interpretability. We provide deeper explanations of some safety-critical capabilities like cybersecurity exploitation, deception, autonomous replication, and situational awareness, alongside concerning propensities like power-seeking and scheming. The review explores how these evaluation methods integrate into governance frameworks to translate results into concrete development decisions. We also highlight challenges to safety evaluations - proving absence of capabilities, potential model sandbagging, and incentives for "safetywashing" - while identifying promising research directions. By synthesizing scattered resources, this literature review aims to provide a central reference point for understanding AI safety evaluations.