# From Silos to Systems: Process-Oriented Hazard Analysis for AI Systems

arXiv: https://arxiv.org/abs/2410.22526 · HTML: https://mlsanity.com/paper/2410.22526/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 62 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Builds on (most cited first, 14 of 14)

- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [The Values Encoded in Machine Learning Research](https://mlsanity.com/paper/2106.15590/) (2021)
- [Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support](https://mlsanity.com/paper/2112.05675/) (2021)
- [The Gradient of Generative AI Release: Methods and Considerations](https://mlsanity.com/paper/2302.04844/) (2023)
- [Typology of Risks of Generative Text-to-Image Models](https://mlsanity.com/paper/2307.05543/) (2023)
- [Dislocated Accountabilities in the AI Supply Chain: Modularity and Developers' Notions of Responsibility](https://mlsanity.com/paper/2209.09780/) (2022)
- [From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML](https://mlsanity.com/paper/2210.03535/) (2022)
- [Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems](https://mlsanity.com/paper/2101.09385/) (2021)
- [Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Development](https://mlsanity.com/paper/2110.02932/) (2021)
- [Current and Near-Term AI as a Potential Existential Risk Factor](https://mlsanity.com/paper/2209.10604/) (2022)
- [Putting AI Ethics into Practice: The Hourglass Model of Organizational AI Governance](https://mlsanity.com/paper/2206.00335/) (2022)
- [On the Challenges of using Reinforcement Learning in Precision Drug Dosing: Delay and Prolongedness of Action Effects](https://mlsanity.com/paper/2301.00512/) (2023)
- [An evidence-based methodology for human rights impact assessment (HRIA) in the development of AI data-intensive systems](https://mlsanity.com/paper/2407.20951/) (2024)

## Built on by (most cited first, 6 of 6)

- [From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systems](https://mlsanity.com/paper/2502.07974/) (2025)
- [Systematic Hazard Analysis for Frontier AI using STPA](https://mlsanity.com/paper/2506.01782/) (2025)
- [Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms](https://mlsanity.com/paper/2510.10339/) (2025)
- [Compiling Sufficient Governance Context from Declared Losses and Reachable States: Exact Observation-Contract Synthesis with Cardinality and Cost Objectives](https://mlsanity.com/paper/2609.26016/) (2026)
- [AI Assurance in UK Defence: Challenges in Operationalising JSP 936](https://mlsanity.com/paper/2606.09414/) (2026)
- [Quantifying System-Level Harms from AI Adoption in Complex Sociotechnical Systems](https://mlsanity.com/paper/2608.23906/) (2026)

## Abstract (arXiv)

To effectively address potential harms from Artificial Intelligence (AI) systems, it is essential to identify and mitigate system-level hazards. Current analysis approaches focus on individual components of an AI system, like training data or models, in isolation, overlooking hazards from component interactions or how they are situated within a company's development process. To this end, we draw from the established field of system safety, which considers safety as an emergent property of the entire system. In this work, we translate System Theoretic Process Analysis (STPA) - a recognized system safety framework - for analyzing AI development and operation processes. We focus on systems that rely on machine learning algorithms and conduct STPA on three case studies involving linear regression, reinforcement learning, and transformer-based generative models. Our analysis explored how STPA's control and system-theoretic perspectives apply to AI systems and whether unique AI traits - such as model opacity, capability uncertainty, and output complexity - necessitate modifications to the framework. We find that the key concepts and steps of conducting an STPA apply to AI systems but require targeted adaptations to address AI-specific challenges that arise to differing degrees across three case studies. We present the Process-oriented Hazard Analysis for AI Systems (PHASE) as a guideline that adapts STPA concepts for AI. Applying and interpreting STPA using the PHASE guidelines enables four key affordances for analysts responsible for managing AI system harms: 1) detection of system-level hazards, including those from accumulation of disparate issues; 2) explicit acknowledgment of social factors contributing to algorithmic harms; 3) creation of traceable accountability chains between harms and those who can mitigate them; and 4) ongoing monitoring and mitigation of new hazards.