mlsanityData to 7 Oct 2026

From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML

arXiv 2210.03535 ↗First posted 6 Oct 2022cs.HC cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Inappropriate design and deployment of machine learning (ML) systems leads to negative downstream social and ethical impact -- described here as social and ethical risks -- for users, society and the environment. Despite the growing need to regulate ML systems, current processes for assessing and mitigating risks are disjointed and inconsistent. We interviewed 30 industry practitioners on their current social and ethical risk management practices, and collected their first reactions on adapting safety engineering frameworks into their practice -- namely, System Theoretic Process Analysis (STPA) and Failure Mode and Effects Analysis (FMEA). Our findings suggest STPA/FMEA can provide appropriate structure toward social and ethical risk assessment and mitigation processes. However, we also find nontrivial challenges in integrating such frameworks in the fast-paced culture of the ML industry. We call on the ML research community to strengthen existing frameworks and assess their efficacy, ensuring that ML systems are safer for all people.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 43 of its 140 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

43 of 140 references.

References · 43

  1. 2018Datasheets for Datasetscited by 1,371
  2. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  3. 2018Model Cards for Model Reportingcited by 1,068
  4. 2017Algorithmic decision making and the cost of fairnesscited by 402
  5. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
  6. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  7. 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
  8. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  9. 2018Manipulating and Measuring Model Interpretabilitycited by 256
  10. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 251
  11. 2019A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cyclecited by 192
  12. 2020Towards Debiasing Sentence Representationscited by 170
31 more
  1. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 150
  2. 2019Towards a Critical Race Methodology in Algorithmic Fairnesscited by 124
  3. 2021Expanding Explainability: Towards Social Transparency in AI systemscited by 124
  4. 2019Artificial Intelligence: the global landscape of ethics guidelinescited by 107
  5. 2021Re-imagining Algorithmic Fairness in India and Beyondcited by 102
  6. 2020Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligencecited by 100
  7. 2022The Fallacy of AI Functionalitycited by 88
  8. 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84
  9. 2021Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Supportcited by 61
  10. 2020"What We Can't Measure, We Can't Understand": Challenges to Demographic Data Procurement in the Pursuit of Fairnesscited by 56
  11. 2021Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Processcited by 56
  12. 2021Mandoline: Model Evaluation under Distribution Shiftcited by 55
  13. 2021Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffscited by 48
  14. 2022Evaluation Gaps in Machine Learning Practicecited by 48
  15. 2021Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviorscited by 47
  16. 2022Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learningcited by 47
  17. 2020Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practicescited by 38
  18. 2021Mitigating harm in language models with conditional-likelihood filtrationcited by 38
  19. 2020Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamicscited by 31
  20. 2022Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethicscited by 28
  21. 2017Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Taskcited by 24
  22. 2019Quantifying Model Complexity via Functional Decomposition for Better Post-Hoc Interpretabilitycited by 22
  23. 2021Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agencycited by 22
  24. 2022System Safety and Artificial Intelligencecited by 18
  25. 2019Defining AI in Policy versus Practicecited by 15
  26. 2022German AI Start-Ups and AI Ethics: Using A Social Practice Lens for Assessing and Implementing Socio-Technical Innovationcited by 11
  27. 2021Simulated Adversarial Testing of Face Recognition Modelscited by 10
  28. 2022The Algorithmic Imprintcited by 10
  29. 2020On Consequentialism and Fairnesscited by 5
  30. 2021Thinking Beyond Distributions in Testing Machine Learned Modelscited by 5
  31. 2022Exploring Opportunities in Usable Hazard Analysis Processes for AI Engineeringcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 9; 2025: 5; 2026: 5.

26 papers.

Sort:

By area:

  1. 2023Model evaluation for extreme riskscited by 167
  2. 2024Visibility into AI Agentscited by 77
  3. 2025Why human-AI relationships need socioaffective alignmentcited by 47
  4. 2025Toward an Evaluation Science for Generative AI Systemscited by 42
  5. 2026Towards Verifiably Safe Tool Use for LLM Agentscited by 34
  6. 2024Farsight: Fostering Responsible AI Awareness During AI Application Prototypingcited by 28
  7. 2023A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitionerscited by 22
  8. 2024A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harmscited by 19
  9. 2024Holistic Safety and Responsibility Evaluations of Advanced AI Modelscited by 18
  10. 2024Harm Amplification in Text-to-Image Modelscited by 14
  11. 2023Deep Learning Safety Concerns in Automated Driving Perceptioncited by 10
  12. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitationscited by 10
14 more
  1. 2024Private Yet Social: How LLM Chatbots Support and Challenge Eating Disorder Recoverycited by 8
  2. 2025From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systemscited by 7
  3. 2024From Silos to Systems: Process-Oriented Hazard Analysis for AI Systemscited by 6
  4. 2023A Systematic Literature Review of Human-Centered, Ethical, and Responsible AIcited by 5
  5. 2022System Safety Engineering for Social and Ethical ML Risks: A Case Studycited by 2
  6. 2023Concrete Safety for ML Problems: System Safety for ML Development and Assessmentcited by 2
  7. 2023Dual Governance: The intersection of centralized regulation and crowdsourced safety mechanisms for Generative AIcited by 2
  8. 2025Talking About the Assumption in the Roomcited by 2
  9. 2025What Is AI Safety? What Do We Want It to Be?cited by 2
  10. 2024Using Quality Attribute Scenarios for ML Model Test Case Generationcited by 0
  11. 2026Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AIcited by 0
  12. 2026Studying People to Study AI: Expert Perspectives on the Epistemic Fit and Barriers of Human Research in AI Safety & Ethicscited by 0
  13. 2026What We Know about Responsible AI Practices in Industry: A Half Decade of Empirical Researchcited by 0
  14. 2026Quantifying System-Level Harms from AI Adoption in Complex Sociotechnical Systemscited by 0