# Risk assessment at AGI companies: A review of popular risk assessment techniques from other safety-critical industries

arXiv: https://arxiv.org/abs/2307.08823 · HTML: https://mlsanity.com/paper/2307.08823/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2023.

## Built on by (most cited first, 19 of 19)

- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Evil Geniuses: Delving into the Safety of LLM-based Agents](https://mlsanity.com/paper/2311.11855/) (2023)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Visibility into AI Agents](https://mlsanity.com/paper/2401.13138/) (2024)
- [Managing extreme AI risks amid rapid progress](https://mlsanity.com/paper/2310.17688/) (2023)
- [Mapping the Ethics of Generative AI: A Comprehensive Scoping Review](https://mlsanity.com/paper/2402.08323/) (2024)
- [A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms](https://mlsanity.com/paper/2407.01294/) (2024)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [Trust AI Regulation? Discerning users are vital to build trust and effective AI regulation](https://mlsanity.com/paper/2403.09510/) (2024)
- [A Frontier AI Risk Management Framework: Bridging the Gap Between Current AI Practices and Established Risk Management](https://mlsanity.com/paper/2502.06656/) (2025)
- [The Role of Governments in Increasing Interconnected Post-Deployment Monitoring of AI](https://mlsanity.com/paper/2410.04931/) (2024)
- [Systematic Hazard Analysis for Frontier AI using STPA](https://mlsanity.com/paper/2506.01782/) (2025)
- [Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation](https://mlsanity.com/paper/2503.04299/) (2025)
- [Overview of Risk Assessment and Management for Intelligent Systems under the AI Act and Beyond](https://mlsanity.com/paper/2607.02197/) (2026)
- [Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems](https://mlsanity.com/paper/2410.23472/) (2024)
- [What Is AI Safety? What Do We Want It to Be?](https://mlsanity.com/paper/2505.02313/) (2025)
- [Taxonomy to Regulation: A (Geo)Political Taxonomy for AI Risks and Regulatory Measures in the EU AI Act](https://mlsanity.com/paper/2404.11476/) (2024)
- [Dimensional Characterization and Pathway Modeling for Catastrophic AI Risks](https://mlsanity.com/paper/2508.06411/) (2025)

## Abstract (arXiv)

Companies like OpenAI, Google DeepMind, and Anthropic have the stated goal of building artificial general intelligence (AGI) - AI systems that perform as well as or better than humans on a wide variety of cognitive tasks. However, there are increasing concerns that AGI would pose catastrophic risks. In light of this, AGI companies need to drastically improve their risk management practices. To support such efforts, this paper reviews popular risk assessment techniques from other safety-critical industries and suggests ways in which AGI companies could use them to assess catastrophic risks from AI. The paper discusses three risk identification techniques (scenario analysis, fishbone method, and risk typologies and taxonomies), five risk analysis techniques (causal mapping, Delphi technique, cross-impact analysis, bow tie analysis, and system-theoretic process analysis), and two risk evaluation techniques (checklists and risk matrices). For each of them, the paper explains how they work, suggests ways in which AGI companies could use them, discusses their benefits and limitations, and makes recommendations. Finally, the paper discusses when to conduct risk assessments, when to use which technique, and how to use any of them. The reviewed techniques will be obvious to risk management professionals in other industries. And they will not be sufficient to assess catastrophic risks from AI. However, AGI companies should not skip the straightforward step of reviewing best practices from other industries.