# Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems

arXiv: https://arxiv.org/abs/1806.07552 · HTML: https://mlsanity.com/paper/1806.07552/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2018.

## Built on by (most cited first, 30 of 48)

- [Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs](https://mlsanity.com/paper/2101.09824/) (2021)
- ["Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction](https://mlsanity.com/paper/2210.03735/) (2022)
- [TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems](https://mlsanity.com/paper/2506.04133/) (2025)
- [How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations](https://mlsanity.com/paper/2101.08758/) (2021)
- [Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contexts](https://mlsanity.com/paper/2201.10295/) (2022)
- [The Who in XAI: How AI Background Shapes Perceptions of AI Explanations](https://mlsanity.com/paper/2107.13509/) (2021)
- [Trust in AI: Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2403.14680/) (2024)
- [Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI](https://mlsanity.com/paper/2302.00799/) (2023)
- [Question-Driven Design Process for Explainable AI User Experiences](https://mlsanity.com/paper/2104.03483/) (2021)
- [The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems](https://mlsanity.com/paper/2602.17753/) (2026)
- [Fast Hierarchical Games for Image Explanations](https://mlsanity.com/paper/2104.06164/) (2021)
- [Directive Explanations for Actionable Explainability in Machine Learning Applications](https://mlsanity.com/paper/2102.02671/) (2021)
- [From Black Boxes to Conversations: Incorporating XAI in a Conversational Agent](https://mlsanity.com/paper/2209.02552/) (2022)
- [How should AI decisions be explained? Requirements for Explanations from the Perspective of European Law](https://mlsanity.com/paper/2404.12762/) (2024)
- [A Comprehensive Review on Financial Explainable AI](https://mlsanity.com/paper/2309.11960/) (2023)
- [How to choose an Explainability Method? Towards a Methodical Implementation of XAI in Practice](https://mlsanity.com/paper/2107.04427/) (2021)
- [A Survey on XAI for 5G and Beyond Security: Technical Aspects, Challenges and Research Directions](https://mlsanity.com/paper/2204.12822/) (2022)
- [AI Explainability 360: Impact and Design](https://mlsanity.com/paper/2109.12151/) (2021)
- [On the Safety of Interpretable Machine Learning: A Maximum Deviation Approach](https://mlsanity.com/paper/2211.01498/) (2022)
- [Explainable Artificial Intelligence (XAI): An Engineering Perspective](https://mlsanity.com/paper/2101.03613/) (2021)
- [SHAP-XRT: The Shapley Value Meets Conditional Independence Testing](https://mlsanity.com/paper/2207.07038/) (2022)
- [What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building Blocks](https://mlsanity.com/paper/2403.12730/) (2024)
- [On Two XAI Cultures: A Case Study of Non-technical Explanations in Deployed AI System](https://mlsanity.com/paper/2112.01016/) (2021)
- [A Means-End Account of Explainable Artificial Intelligence](https://mlsanity.com/paper/2208.04638/) (2022)
- [Seven challenges for harmonizing explainability requirements](https://mlsanity.com/paper/2108.05390/) (2021)
- [Regulation Games for Trustworthy Machine Learning](https://mlsanity.com/paper/2402.03540/) (2024)
- [Sufficient and Necessary Explanations (and What Lies in Between)](https://mlsanity.com/paper/2409.20427/) (2024)
- [A Mechanistic Explanatory Strategy for XAI](https://mlsanity.com/paper/2411.01332/) (2024)
- [On the Design and Evaluation of Human-centered Explainable AI Systems: A Systematic Review and Taxonomy](https://mlsanity.com/paper/2510.12201/) (2025)
- [The Case Against Explainability](https://mlsanity.com/paper/2305.12167/) (2023)

## Abstract (arXiv)

Several researchers have argued that a machine learning system's interpretability should be defined in relation to a specific agent or task: we should not ask if the system is interpretable, but to whom is it interpretable. We describe a model intended to help answer this question, by identifying different roles that agents can fulfill in relation to the machine learning system. We illustrate the use of our model in a variety of scenarios, exploring how an agent's role influences its goals, and the implications for defining interpretability. Finally, we make suggestions for how our model could be useful to interpretability researchers, system developers, and regulatory bodies auditing machine learning systems.