# On Two XAI Cultures: A Case Study of Non-technical Explanations in Deployed AI System

arXiv: https://arxiv.org/abs/2112.01016 · HTML: https://mlsanity.com/paper/2112.01016/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 55 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 14 of 14)

- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Actionable Recourse in Linear Classification](https://mlsanity.com/paper/1809.06514/) (2018)
- [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019)
- [Explaining Explanations in AI](https://mlsanity.com/paper/1811.01439/) (2018)
- [What do we need to build explainable AI systems for the medical domain?](https://mlsanity.com/paper/1712.09923/) (2017)
- [Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches](https://mlsanity.com/paper/1912.05100/) (2019)
- [Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences](https://mlsanity.com/paper/1712.00547/) (2017)
- [How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation](https://mlsanity.com/paper/1802.00682/) (2018)
- [Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems](https://mlsanity.com/paper/1806.07552/) (2018)
- [Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creators](https://mlsanity.com/paper/1901.06417/) (2019)
- [A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning](https://mlsanity.com/paper/1801.05075/) (2018)
- [Towards a Characterization of Explainable Systems](https://mlsanity.com/paper/1902.03096/) (2019)

## Built on by (most cited first, 5 of 5)

- [LLMs for XAI: Future Directions for Explaining Explanations](https://mlsanity.com/paper/2405.06064/) (2024)
- [Explingo: Explaining AI Predictions using Large Language Models](https://mlsanity.com/paper/2412.05145/) (2024)
- [Lessons from Usable ML Deployments and Application to Wind Turbine Monitoring](https://mlsanity.com/paper/2312.02859/) (2023)
- [Pyreal: A Framework for Interpretable ML Explanations](https://mlsanity.com/paper/2312.13084/) (2023)
- [Feature-Guided Neighbor Selection for Non-Expert Evaluation of Model Predictions](https://mlsanity.com/paper/2507.06029/) (2025)

## Abstract (arXiv)

Explainable AI (XAI) research has been booming, but the question "To whom are we making AI explainable?" is yet to gain sufficient attention. Not much of XAI is comprehensible to non-AI experts, who nonetheless, are the primary audience and major stakeholders of deployed AI systems in practice. The gap is glaring: what is considered "explained" to AI-experts versus non-experts are very different in practical scenarios. Hence, this gap produced two distinct cultures of expectations, goals, and forms of XAI in real-life AI deployments. We advocate that it is critical to develop XAI methods for non-technical audiences. We then present a real-life case study, where AI experts provided non-technical explanations of AI decisions to non-technical stakeholders, and completed a successful deployment in a highly regulated industry. We then synthesize lessons learned from the case, and share a list of suggestions for AI experts to consider when explaining AI decisions to non-technical stakeholders.