# The Normalization of Deviance in AI Development

arXiv: https://arxiv.org/abs/2609.05749 · HTML: https://mlsanity.com/paper/2609.05749/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 57 references that are in the map.
- First posted on arXiv 4 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies](https://mlsanity.com/paper/2601.11699/) (2026)
- [International Institutions for Advanced AI](https://mlsanity.com/paper/2307.04699/) (2023)
- [How Do AI Companies "Fine-Tune" Policy? Examining Regulatory Capture in AI Governance](https://mlsanity.com/paper/2410.13042/) (2024)
- [Mapping Technical Safety Research at AI Companies: A literature review and incentives analysis](https://mlsanity.com/paper/2409.07878/) (2024)
- [Beyond Benchmarks: On The False Promise of AI Regulation](https://mlsanity.com/paper/2501.15693/) (2025)
- [The 2025 OpenAI Preparedness Framework does not guarantee any AI risk mitigation practices: a proof-of-concept for affordance analyses of AI safety policies](https://mlsanity.com/paper/2509.24394/) (2025)
- [Enabling Frontier Lab Collaboration to Mitigate AI Safety Risks](https://mlsanity.com/paper/2511.08631/) (2025)
- [From Disclosure to Self-Referential Opacity: Six Dimensions of Strain in Current AI Governance](https://mlsanity.com/paper/2604.14070/) (2026)

## Abstract (arXiv)

Work on the risks of artificial intelligence has focused predominantly on capability risk: the danger that systems become too powerful, too autonomous, or too misaligned with human values. Far less attention has been paid to the organizational level---to whether the institutions building these systems are themselves predisposed to drift toward failure. This paper argues that they are. Regardless of how capable AI systems become, the organizations building them face the same structural dynamics that preceded past major technological disasters. Drawing on case studies of the Space Shuttle Challenger, the Three Mile Island accident, and the Boeing 737 MAX crashes, this paper identifies the common structural mechanisms preceding each failure and maps them onto contemporary AI development. The findings suggest that existing safety infrastructure may provide less protection than it appears, as organizations can complete safety processes in full compliance and still produce catastrophic outcomes. The pre-disaster period of AI development is still underway; the purpose of this paper is to make these dynamics legible while they can still be interrupted.