# Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI

arXiv: https://arxiv.org/abs/2607.14353 · HTML: https://mlsanity.com/paper/2607.14353/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 155 references that are in the map.
- First posted on arXiv 15 Jul 2026.

## Builds on (most cited first, 30 of 39)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure](https://mlsanity.com/paper/2010.13561/) (2020)
- [Power to the People? Opportunities and Challenges for Participatory AI](https://mlsanity.com/paper/2209.07572/) (2022)
- [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022)
- [The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards](https://mlsanity.com/paper/1805.03677/) (2018)
- [The Fallacy of AI Functionality](https://mlsanity.com/paper/2206.09511/) (2022)
- [What Does Explainable AI Really Mean? A New Conceptualization of Perspectives](https://mlsanity.com/paper/1710.00794/) (2017)
- [International AI Safety Report](https://mlsanity.com/paper/2501.17805/) (2025)
- [Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning](https://mlsanity.com/paper/2202.05338/) (2022)
- [Understanding accountability in algorithmic supply chains](https://mlsanity.com/paper/2304.14749/) (2023)
- [Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems](https://mlsanity.com/paper/2102.04201/) (2021)
- [Understanding Machine Learning Practitioners' Data Documentation Perceptions, Needs, Challenges, and Desiderata](https://mlsanity.com/paper/2206.02923/) (2022)
- [From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML](https://mlsanity.com/paper/2210.03535/) (2022)
- [Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems](https://mlsanity.com/paper/2101.09385/) (2021)
- [Concrete Problems in AI Safety, Revisited](https://mlsanity.com/paper/2401.10899/) (2023)
- [Chaos Engineering](https://mlsanity.com/paper/1702.05843/) (2017)
- [System Safety and Artificial Intelligence](https://mlsanity.com/paper/2202.09292/) (2022)
- [A Silicon Valley Love Triangle: Hiring Algorithms, Pseudo-Science, and the Quest for Auditability](https://mlsanity.com/paper/2106.12403/) (2021)
- [Algorithms as Social-Ecological-Technological Systems: an Environmental Justice Lens on Algorithmic Audits](https://mlsanity.com/paper/2305.05733/) (2023)
- [Beyond the ML Model: Applying Safety Engineering Frameworks to Text-to-Image Development](https://mlsanity.com/paper/2307.10312/) (2023)
- [Improving governance outcomes through AI documentation: Bridging theory and practice](https://mlsanity.com/paper/2409.08960/) (2024)
- [Tackling COVID-19 through Responsible AI Innovation: Five Steps in the Right Direction](https://mlsanity.com/paper/2008.06755/) (2020)
- [AI Development for the Public Interest: From Abstraction Traps to Sociotechnical Risks](https://mlsanity.com/paper/2102.04255/) (2021)
- [Beyond Model Interpretability: Socio-Structural Explanations in Machine Learning](https://mlsanity.com/paper/2409.03632/) (2024)

## Abstract (arXiv)

As automated decision-making and data-driven technologies pervade society and are used to manage consequential outcomes, understanding the technology's capabilities, limitations, and attendant risks in context requires analysis of full sociotechnical systems. Sociotechnical analysis of risks in highly complex systems provides clear lessons for the design and evaluation of AI systems, transcending a technical focus on reliable or "responsibly designed" components to understand risks at a systems level. Human-made catastrophes have been studied for decades because of the severity of these events: consider Chernobyl, Three Mile Island, Fukushima-Daiichi, Bhopal, the Challenger disaster. A common misconception is that these kinds of events are freak accidents, resulting from the inherently unforeseeable interactions in complex systems. Closer examination reveals that the risks and hazards were well-known beforehand but not acted upon due to social structural, political and economic factors. We outline several areas where the development and use of AI can benefit from learning these unlearned lessons: improved risk perception, communication, and analysis at the organizational level; traceability of requirements and responsibilities; and holistic approaches to responsibility and safety that include social and organizational dynamics as first-order engineering concerns. For each area, we offer concrete unlearned lessons and exemplify how they led to failure in prior accidents as well as examples of how these lessons remain unlearned for modern computing systems, particularly AI.