# Risk thresholds for frontier AI

arXiv: https://arxiv.org/abs/2406.14713 · HTML: https://mlsanity.com/paper/2406.14713/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2024.

## Built on by (most cited first, 9 of 9)

- [A sketch of an AI control safety case](https://mlsanity.com/paper/2501.17315/) (2025)
- [Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agents](https://mlsanity.com/paper/2502.11355/) (2025)
- [Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems](https://mlsanity.com/paper/2410.23472/) (2024)
- [Technical Requirements for Halting Dangerous AI Activities](https://mlsanity.com/paper/2507.09801/) (2025)
- [Mechanism Design for Alignment and Control](https://mlsanity.com/paper/2609.01595/) (2026)
- [Quantifying detection rates for dangerous capabilities: a theoretical model of dangerous capability evaluations](https://mlsanity.com/paper/2412.15433/) (2024)
- [What Makes an Evaluation Useful? Common Pitfalls and Best Practices](https://mlsanity.com/paper/2503.23424/) (2025)
- [Harmonizing AI Safety Thresholds](https://mlsanity.com/paper/2607.16112/) (2026)
- [Silent Revision: Measuring Undisclosed Change in the Safety Frameworks of Frontier AI Developers](https://mlsanity.com/paper/2609.08789/) (2026)

## Abstract (arXiv)

Frontier artificial intelligence (AI) systems could pose increasing risks to public safety and security. But what level of risk is acceptable? One increasingly popular approach is to define capability thresholds, which describe AI capabilities beyond which an AI system is deemed to pose too much risk. A more direct approach is to define risk thresholds that simply state how much risk would be too much. For instance, they might state that the likelihood of cybercriminals using an AI system to cause X amount of economic damage must not increase by more than Y percentage points. The main upside of risk thresholds is that they are more principled than capability thresholds, but the main downside is that they are more difficult to evaluate reliably. For this reason, we currently recommend that companies (1) define risk thresholds to provide a principled foundation for their decision-making, (2) use these risk thresholds to help set capability thresholds, and then (3) primarily rely on capability thresholds to make their decisions. Regulators should also explore the area because, ultimately, they are the most legitimate actors to define risk thresholds. If AI risk estimates become more reliable, risk thresholds should arguably play an increasingly direct role in decision-making.