# Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences

arXiv: https://arxiv.org/abs/1712.00547 · HTML: https://mlsanity.com/paper/1712.00547/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2017.

## Built on by (most cited first, 30 of 85)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [Human-Centered Explainable AI (XAI): From Algorithms to User Experiences](https://mlsanity.com/paper/2110.10790/) (2021)
- [What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research](https://mlsanity.com/paper/2102.07817/) (2021)
- [If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniques](https://mlsanity.com/paper/2103.01035/) (2021)
- [GraphSVX: Shapley Value Explanations for Graph Neural Networks](https://mlsanity.com/paper/2104.10482/) (2021)
- ["Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction](https://mlsanity.com/paper/2210.03735/) (2022)
- [How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations](https://mlsanity.com/paper/2101.08758/) (2021)
- [Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI](https://mlsanity.com/paper/2206.10847/) (2022)
- [Transitioning to human interaction with AI systems: New challenges and opportunities for HCI professionals to enable human-centered AI](https://mlsanity.com/paper/2105.05424/) (2021)
- [Trust in AI: Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2403.14680/) (2024)
- [A Study of Automatic Metrics for the Evaluation of Natural Language Explanations](https://mlsanity.com/paper/2103.08545/) (2021)
- [Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory](https://mlsanity.com/paper/2205.05057/) (2022)
- [ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing](https://mlsanity.com/paper/2305.09770/) (2023)
- [Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey](https://mlsanity.com/paper/2108.09003/) (2021)
- [Levels of explainable artificial intelligence for human-aligned conversational explanations](https://mlsanity.com/paper/2107.03178/) (2021)
- [Tell Me a Story! Narrative-Driven XAI with Large Language Models](https://mlsanity.com/paper/2309.17057/) (2023)
- [Contrastive Explanations of Plans Through Model Restrictions](https://mlsanity.com/paper/2103.15575/) (2021)
- [A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations](https://mlsanity.com/paper/2105.02626/) (2021)
- [XAI Handbook: Towards a Unified Framework for Explainable AI](https://mlsanity.com/paper/2105.06677/) (2021)
- [Explainability Is in the Mind of the Beholder: Establishing the Foundations of Explainable Artificial Intelligence](https://mlsanity.com/paper/2112.14466/) (2021)
- [Towards Compositional Interpretability for XAI](https://mlsanity.com/paper/2406.17583/) (2024)
- [Establishing and Evaluating Trustworthy AI: Overview and Research Challenges](https://mlsanity.com/paper/2411.09973/) (2024)
- [How should AI decisions be explained? Requirements for Explanations from the Perspective of European Law](https://mlsanity.com/paper/2404.12762/) (2024)
- [GraphXAIN: Narratives to Explain Graph Neural Networks](https://mlsanity.com/paper/2411.02540/) (2024)
- [Mind the Gap! Bridging Explainable Artificial Intelligence and Human Understanding with Luhmann's Functional Theory of Communication](https://mlsanity.com/paper/2302.03460/) (2023)
- [AdViCE: Aggregated Visual Counterfactual Explanations for Machine Learning Model Validation](https://mlsanity.com/paper/2109.05629/) (2021)
- [SoK: Modeling Explainability in Security Analytics for Interpretability, Trustworthiness, and Usability](https://mlsanity.com/paper/2210.17376/) (2022)
- [Why do explanations fail? A typology and discussion on failures in XAI](https://mlsanity.com/paper/2405.13474/) (2024)
- [Beyond Model Interpretability: Socio-Structural Explanations in Machine Learning](https://mlsanity.com/paper/2409.03632/) (2024)

## Abstract (arXiv)

In his seminal book `The Inmates are Running the Asylum: Why High-Tech Products Drive Us Crazy And How To Restore The Sanity' [2004, Sams Indianapolis, IN, USA], Alan Cooper argues that a major reason why software is often poorly designed (from a user perspective) is that programmers are in charge of design decisions, rather than interaction designers. As a result, programmers design software for themselves, rather than for their target audience, a phenomenon he refers to as the `inmates running the asylum'. This paper argues that explainable AI risks a similar fate. While the re-emergence of explainable AI is positive, this paper argues most of us as AI researchers are building explanatory agents for ourselves, rather than for the intended users. But explainable AI is more likely to succeed if researchers and practitioners understand, adopt, implement, and improve models from the vast and valuable bodies of research in philosophy, psychology, and cognitive science, and if evaluation of these models is focused more on people than on technology. From a light scan of literature, we demonstrate that there is considerable scope to infuse more results from the social and behavioural sciences into explainable AI, and present some key results from these fields that are relevant to explainable AI.