# Quantifying detection rates for dangerous capabilities: a theoretical model of dangerous capability evaluations

arXiv: https://arxiv.org/abs/2412.15433 · HTML: https://mlsanity.com/paper/2412.15433/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Builds on (most cited first, 21 of 21)

- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [An Overview of Catastrophic AI Risks](https://mlsanity.com/paper/2306.12001/) (2023)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)
- [Evaluating Language-Model Agents on Realistic Autonomous Tasks](https://mlsanity.com/paper/2312.11671/) (2023)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?](https://mlsanity.com/paper/2407.21792/) (2024)
- [Sabotage Evaluations for Frontier Models](https://mlsanity.com/paper/2410.21514/) (2024)
- [Open Problems in Technical AI Governance](https://mlsanity.com/paper/2407.14981/) (2024)
- [The Role of Cooperation in Responsible AI Development](https://mlsanity.com/paper/1907.04534/) (2019)
- [Safety case template for frontier AI: A cyber inability argument](https://mlsanity.com/paper/2411.08088/) (2024)
- [Safety cases for frontier AI](https://mlsanity.com/paper/2410.21572/) (2024)
- [Why and How Governments Should Monitor AI Development](https://mlsanity.com/paper/2108.12427/) (2021)
- [Coordinated pausing: An evaluation-based coordination scheme for frontier AI developers](https://mlsanity.com/paper/2310.00374/) (2023)
- [Voluntary safety commitments provide an escape from over-regulation in AI development](https://mlsanity.com/paper/2104.03741/) (2021)
- [Risk thresholds for frontier AI](https://mlsanity.com/paper/2406.14713/) (2024)
- [Social Diversity Reduces the Complexity and Cost of Fostering Fairness](https://mlsanity.com/paper/2211.10517/) (2022)
- [An International Consortium for Evaluations of Societal-Scale Risks from Advanced AI](https://mlsanity.com/paper/2310.14455/) (2023)
- [Who is leading in AI? An analysis of industry AI research](https://mlsanity.com/paper/2312.00043/) (2023)
- [Analyzing Probabilistic Methods for Evaluating Agent Capabilities](https://mlsanity.com/paper/2409.16125/) (2024)

## Built on by (most cited first, 1 of 1)

- [Distributional AGI Safety](https://mlsanity.com/paper/2512.16856/) (2025)

## Abstract (arXiv)

We present a quantitative model for tracking dangerous AI capabilities over time. Our goal is to help the policy and research community visualise how dangerous capability testing can give us an early warning about approaching AI risks. We first use the model to provide a novel introduction to dangerous capability testing and how this testing can directly inform policy. Decision makers in AI labs and government often set policy that is sensitive to the estimated danger of AI systems, and may wish to set policies that condition on the crossing of a set threshold for danger. The model helps us to reason about these policy choices. We then run simulations to illustrate how we might fail to test for dangerous capabilities. To summarise, failures in dangerous capability testing may manifest in two ways: higher bias in our estimates of AI danger, or larger lags in threshold monitoring. We highlight two drivers of these failure modes: uncertainty around dynamics in AI capabilities and competition between frontier AI labs. Effective AI policy demands that we address these failure modes and their drivers. Even if the optimal targeting of resources is challenging, we show how delays in testing can harm AI policy. We offer preliminary recommendations for building an effective testing ecosystem for dangerous capabilities and advise on a research agenda.