# System Safety Engineering for Social and Ethical ML Risks: A Case Study

arXiv: https://arxiv.org/abs/2211.04602 · HTML: https://mlsanity.com/paper/2211.04602/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 45 references that are in the map.
- First posted on arXiv 8 Nov 2022.

## Builds on (most cited first, 12 of 12)

- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support](https://mlsanity.com/paper/2112.05675/) (2021)
- [Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning](https://mlsanity.com/paper/2202.05338/) (2022)
- [Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics](https://mlsanity.com/paper/2005.07572/) (2020)
- [From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML](https://mlsanity.com/paper/2210.03535/) (2022)
- [This Thing Called Fairness: Disciplinary Confusion Realizing a Value in Technology](https://mlsanity.com/paper/1909.11869/) (2019)
- [A Hierarchy of Limitations in Machine Learning](https://mlsanity.com/paper/2002.05193/) (2020)
- [Extending the Machine Learning Abstraction Boundary: A Complex Systems Approach to Incorporate Societal Context](https://mlsanity.com/paper/2006.09663/) (2020)
- [System Safety and Artificial Intelligence](https://mlsanity.com/paper/2202.09292/) (2022)

## Built on by (most cited first, 2 of 2)

- [From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systems](https://mlsanity.com/paper/2502.07974/) (2025)
- [FairSense: Long-Term Fairness Analysis of ML-Enabled Systems](https://mlsanity.com/paper/2501.01665/) (2025)

## Abstract (arXiv)

Governments, industry, and academia have undertaken efforts to identify and mitigate harms in ML-driven systems, with a particular focus on social and ethical risks of ML components in complex sociotechnical systems. However, existing approaches are largely disjointed, ad-hoc and of unknown effectiveness. Systems safety engineering is a well established discipline with a track record of identifying and managing risks in many complex sociotechnical domains. We adopt the natural hypothesis that tools from this domain could serve to enhance risk analyses of ML in its context of use. To test this hypothesis, we apply a "best of breed" systems safety analysis, Systems Theoretic Process Analysis (STPA), to a specific high-consequence system with an important ML-driven component, namely the Prescription Drug Monitoring Programs (PDMPs) operated by many US States, several of which rely on an ML-derived risk score. We focus in particular on how this analysis can extend to identifying social and ethical risks and developing concrete design-level controls to mitigate them.