# Open Problems in AI Incident Governance

arXiv: https://arxiv.org/abs/2607.05163 · HTML: https://mlsanity.com/paper/2607.05163/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 76 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Builds on (most cited first, 11 of 11)

- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents](https://mlsanity.com/paper/2509.26354/) (2025)
- [From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents](https://mlsanity.com/paper/2606.04990/) (2026)
- [Async Control: Stress-testing Asynchronous Control Measures for LLM Agents](https://mlsanity.com/paper/2512.13526/) (2025)
- [Safety cases for frontier AI](https://mlsanity.com/paper/2410.21572/) (2024)
- [A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage](https://mlsanity.com/paper/2504.21035/) (2025)
- [Deployment Corrections: An incident response framework for frontier AI models](https://mlsanity.com/paper/2310.00328/) (2023)
- [The Role of Governments in Increasing Interconnected Post-Deployment Monitoring of AI](https://mlsanity.com/paper/2410.04931/) (2024)
- [Exploring the Relevance of Data Privacy-Enhancing Technologies for AI Governance Use Cases](https://mlsanity.com/paper/2303.08956/) (2023)
- [AI Harmonics: a human-centric and harms severity-adaptive AI risk assessment framework](https://mlsanity.com/paper/2509.10104/) (2025)

## Built on by (most cited first, 1 of 1)

- [Testing and Evaluation of Agentic AI Systems In Military Command and Control](https://mlsanity.com/paper/2608.20597/) (2026)

## Abstract (arXiv)

AI systems may produce failures after deployment that pre-deployment safety assessments do not anticipate. Managing these failures requires what we refer to as adequate textitAI incident governance, where having good definitions, taxonomies, monitoring practices, reporting mechanisms, and incident analysis is essential. We examine existing frameworks related to AI incident governance by regulatory bodies and independent efforts, and find that while there are frameworks that describe how individual functions can be performed, there is a lack of consistency within the aspects of definitions, classification, monitoring, and reporting. These inconsistencies apply to the types of incident data that is collected and reported, the ways in which they are categorised, and as a result, the depth, representativeness, and accuracy of analysis that can be performed.