# Actionable Guidance for High-Consequence AI Risk Management: Towards Standards Addressing AI Catastrophic Risks

arXiv: https://arxiv.org/abs/2206.08966 · HTML: https://mlsanity.com/paper/2206.08966/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 70 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Builds on (most cited first, 13 of 13)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation](https://mlsanity.com/paper/1802.07228/) (2018)
- [Structured access: an emerging paradigm for safe AI deployment](https://mlsanity.com/paper/2201.05159/) (2022)
- [Choices, Risks, and Reward Reports: Charting Public Policy for Reinforcement Learning Systems](https://mlsanity.com/paper/2202.05716/) (2022)
- [Analysing Results from AI Benchmarks: Key Indicators and How to Obtain Them](https://mlsanity.com/paper/1811.08186/) (2018)

## Built on by (most cited first, 7 of 7)

- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives](https://mlsanity.com/paper/2311.09227/) (2023)
- [Attestable Audits: Verifiable AI Safety Benchmarks Using Trusted Execution Environments](https://mlsanity.com/paper/2506.23706/) (2025)
- [Both eyes open: Vigilant Incentives help Regulatory Markets improve AI Safety](https://mlsanity.com/paper/2303.03174/) (2023)
- [Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems](https://mlsanity.com/paper/2410.23472/) (2024)
- [Agent-Supported Foresight for AI Systemic Risks: AI Agents for Breadth, Experts for Judgment](https://mlsanity.com/paper/2602.08565/) (2026)
- [Governable AI: Provable Safety Under Extreme Threat Models](https://mlsanity.com/paper/2508.20411/) (2025)

## Abstract (arXiv)

Artificial intelligence (AI) systems can provide many beneficial capabilities but also risks of adverse events. Some AI systems could present risks of events with very high or catastrophic consequences at societal scale. The US National Institute of Standards and Technology (NIST) has been developing the NIST Artificial Intelligence Risk Management Framework (AI RMF) as voluntary guidance on AI risk assessment and management for AI developers and others. For addressing risks of events with catastrophic consequences, NIST indicated a need to translate from high level principles to actionable risk management guidance. In this document, we provide detailed actionable-guidance recommendations focused on identifying and managing risks of events with very high or catastrophic consequences, intended as a risk management practices resource for NIST for AI RMF version 1.0 (released in January 2023), or for AI RMF users, or for other AI risk management guidance and standards as appropriate. We also provide our methodology for our recommendations. We provide actionable-guidance recommendations for AI RMF 1.0 on: identifying risks from potential unintended uses and misuses of AI systems; including catastrophic-risk factors within the scope of risk assessments and impact assessments; identifying and mitigating human rights harms; and reporting information on AI risk factors including catastrophic-risk factors. In addition, we provide recommendations on additional issues for a roadmap for later versions of the AI RMF or supplementary publications. These include: providing an AI RMF Profile with supplementary guidance for cutting-edge increasingly multi-purpose or general-purpose AI. We aim for this work to be a concrete risk-management practices contribution, and to stimulate constructive dialogue on how to address catastrophic risks and associated issues in AI standards.