# A taxonomic system for failure cause analysis of open source AI incidents

arXiv: https://arxiv.org/abs/2211.07280 · HTML: https://mlsanity.com/paper/2211.07280/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 19 references that are in the map.
- First posted on arXiv 14 Nov 2022.

## Builds on (most cited first, 6 of 6)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Categorizing Variants of Goodhart's Law](https://mlsanity.com/paper/1803.04585/) (2018)
- [Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database](https://mlsanity.com/paper/2011.08512/) (2020)
- [Artificial Intelligence Safety and Cybersecurity: a Timeline of AI Failures](https://mlsanity.com/paper/1610.07997/) (2016)
- [AI Failures: A Review of Underlying Issues](https://mlsanity.com/paper/2008.04073/) (2020)

## Built on by (most cited first, 9 of 9)

- [RealHarm: A Collection of Real-World Language Model Application Failures](https://mlsanity.com/paper/2504.10277/) (2025)
- [Lessons for Editors of AI Incidents from the AI Incident Database](https://mlsanity.com/paper/2409.16425/) (2024)
- [Incident Analysis for AI Agents](https://mlsanity.com/paper/2508.14231/) (2025)
- [AI Risk Profiles: A Standards Proposal for Pre-Deployment AI Risk Disclosures](https://mlsanity.com/paper/2309.13176/) (2023)
- [The Agent Incident Registry: Toward Preventing Repeated AI Agent Failures](https://mlsanity.com/paper/2609.11030/) (2026)
- [Towards Safer Operations: An Expert-involved Dataset of High-Pressure Gas Incidents for Preventing Future Failures](https://mlsanity.com/paper/2310.12074/) (2023)
- [Dimensional Characterization and Pathway Modeling for Catastrophic AI Risks](https://mlsanity.com/paper/2508.06411/) (2025)
- [Designing escalation criteria for international AI incident response: criteria, triggers, and thresholds](https://mlsanity.com/paper/2604.23183/) (2026)
- [Physiological Noise Augmentation Improves Non-Invasive Brain-to-Speech](https://mlsanity.com/paper/2607.05165/) (2026)

## Abstract (arXiv)

While certain industrial sectors (e.g., aviation) have a long history of mandatory incident reporting complete with analytical findings, the practice of artificial intelligence (AI) safety benefits from no such mandate and thus analyses must be performed on publicly known ``open source'' AI incidents. Although the exact causes of AI incidents are seldom known by outsiders, this work demonstrates how to apply expert knowledge on the population of incidents in the AI Incident Database (AIID) to infer the potential and likely technical causative factors that contribute to reported failures and harms. We present early work on a taxonomic system that covers a cascade of interrelated incident factors, from system goals (nearly always known) to methods / technologies (knowable in many cases) and technical failure causes (subject to expert analysis) of the implicated systems. We pair this ontology structure with a comprehensive classification workflow that leverages expert knowledge and community feedback, resulting in taxonomic annotations grounded by incident data and human expertise.