# Beyond Model Interpretability: Socio-Structural Explanations in Machine Learning

arXiv: https://arxiv.org/abs/2409.03632 · HTML: https://mlsanity.com/paper/2409.03632/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 73 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Builds on (most cited first, 30 of 35)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [On the Robustness of Interpretability Methods](https://mlsanity.com/paper/1806.08049/) (2018)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016)
- [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019)
- [Explaining Explanations in AI](https://mlsanity.com/paper/1811.01439/) (2018)
- [OpenXAI: Towards a Transparent Evaluation of Model Explanations](https://mlsanity.com/paper/2206.11104/) (2022)
- [Towards a Critical Race Methodology in Algorithmic Fairness](https://mlsanity.com/paper/1912.03593/) (2019)
- [Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities](https://mlsanity.com/paper/2111.06420/) (2021)
- [Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences](https://mlsanity.com/paper/1712.00547/) (2017)
- [Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations](https://mlsanity.com/paper/2206.01254/) (2022)
- [The Use and Misuse of Counterfactuals in Ethical Machine Learning](https://mlsanity.com/paper/2102.05085/) (2021)
- [Local Explanations via Necessity and Sufficiency: Unifying Theory and Practice](https://mlsanity.com/paper/2103.14651/) (2021)
- [Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics](https://mlsanity.com/paper/2005.07572/) (2020)
- [Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps](https://mlsanity.com/paper/2005.08874/) (2020)
- [Operationalizing Machine Learning: An Interview Study](https://mlsanity.com/paper/2209.09125/) (2022)
- [Best of both worlds: local and global explanations with human-understandable concepts](https://mlsanity.com/paper/2106.08641/) (2021)
- [Explainable Artificial Intelligence for Process Mining: A General Overview and Application of a Novel Local Explanation Approach for Predictive Process Monitoring](https://mlsanity.com/paper/2009.02098/) (2020)

## Built on by (most cited first, 6 of 6)

- [Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrest](https://mlsanity.com/paper/2504.04243/) (2025)
- [From Causal Discovery to Dynamic Causal Inference in Neural Time Series](https://mlsanity.com/paper/2603.20980/) (2026)
- [Position: Beyond Sensitive Attributes, ML Fairness Should Quantify Structural Injustice via Social Determinants](https://mlsanity.com/paper/2508.08337/) (2025)
- [Beyond Technocratic XAI: The Who, What & How in Explanation Design](https://mlsanity.com/paper/2508.09231/) (2025)
- [Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI](https://mlsanity.com/paper/2607.14353/) (2026)
- [Safe Evolution with Circuit Anchors](https://mlsanity.com/paper/2608.05158/) (2026)

## Abstract (arXiv)

What is it to interpret the outputs of an opaque machine learning model. One approach is to develop interpretable machine learning techniques. These techniques aim to show how machine learning models function by providing either model centric local or global explanations, which can be based on mechanistic interpretations revealing the inner working mechanisms of models or nonmechanistic approximations showing input feature output data relationships. In this paper, we draw on social philosophy to argue that interpreting machine learning outputs in certain normatively salient domains could require appealing to a third type of explanation that we call sociostructural explanation. The relevance of this explanation type is motivated by the fact that machine learning models are not isolated entities but are embedded within and shaped by social structures. Sociostructural explanations aim to illustrate how social structures contribute to and partially explain the outputs of machine learning models. We demonstrate the importance of sociostructural explanations by examining a racially biased healthcare allocation algorithm. Our proposal highlights the need for transparency beyond model interpretability, understanding the outputs of machine learning systems could require a broader analysis that extends beyond the understanding of the machine learning model itself.