# The Challenge of Crafting Intelligible Intelligence

arXiv: https://arxiv.org/abs/1803.04263 · HTML: https://mlsanity.com/paper/1803.04263/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2018.

## Built on by (most cited first, 30 of 47)

- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [Expanding Explainability: Towards Social Transparency in AI systems](https://mlsanity.com/paper/2101.04719/) (2021)
- [Rethinking Interpretability in the Era of Large Language Models](https://mlsanity.com/paper/2402.01761/) (2024)
- [TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations](https://mlsanity.com/paper/2207.04154/) (2022)
- [NICE: An Algorithm for Nearest Instance Counterfactual Explanations](https://mlsanity.com/paper/2104.07411/) (2021)
- [In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making](https://mlsanity.com/paper/2305.07722/) (2023)
- [LLMs for Explainable AI: A Comprehensive Survey](https://mlsanity.com/paper/2504.00125/) (2025)
- [Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI](https://mlsanity.com/paper/2206.10847/) (2022)
- [Explaining a Series of Models by Propagating Shapley Values](https://mlsanity.com/paper/2105.00108/) (2021)
- [On Efficiently Explaining Graph-Based Classifiers](https://mlsanity.com/paper/2106.01350/) (2021)
- [Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory](https://mlsanity.com/paper/2205.05057/) (2022)
- [Nonperturbative renormalization for the neural network-QFT correspondence](https://mlsanity.com/paper/2108.01403/) (2021)
- [A Computational Inflection for Scientific Discovery](https://mlsanity.com/paper/2205.02007/) (2022)
- [On the Influence of Explainable AI on Automation Bias](https://mlsanity.com/paper/2204.08859/) (2022)
- [From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence](https://mlsanity.com/paper/2104.13299/) (2021)
- [HINT: Hierarchical Neuron Concept Explainer](https://mlsanity.com/paper/2203.14196/) (2022)
- [Explainability Is in the Mind of the Beholder: Establishing the Foundations of Explainable Artificial Intelligence](https://mlsanity.com/paper/2112.14466/) (2021)
- [Efficient Explanations With Relevant Sets](https://mlsanity.com/paper/2106.00546/) (2021)
- [InterroLang: Exploring NLP Models and Datasets through Dialogue-based Explanations](https://mlsanity.com/paper/2310.05592/) (2023)
- [Efficient Explanations for Knowledge Compilation Languages](https://mlsanity.com/paper/2107.01654/) (2021)
- [GAM Coach: Towards Interactive and User-centered Algorithmic Recourse](https://mlsanity.com/paper/2302.14165/) (2023)
- [Why is plausibility surprisingly problematic as an XAI criterion?](https://mlsanity.com/paper/2303.17707/) (2023)
- [What Else Do I Need to Know? The Effect of Background Information on Users' Reliance on QA Systems](https://mlsanity.com/paper/2305.14331/) (2023)
- [Mediators: Conversational Agents Explaining NLP Model Behavior](https://mlsanity.com/paper/2206.06029/) (2022)
- [Renormalization in the neural network-quantum field theory correspondence](https://mlsanity.com/paper/2212.11811/) (2022)
- [Understanding Practices around Computational News Discovery Tools in the Domain of Science Journalism](https://mlsanity.com/paper/2311.06864/) (2023)
- [CoXQL: A Dataset for Parsing Explanation Requests in Conversational XAI Systems](https://mlsanity.com/paper/2406.08101/) (2024)
- [A Fuzzy-based Approach to Predict Human Interaction by Functional Near-Infrared Spectroscopy](https://mlsanity.com/paper/2409.17661/) (2024)
- [A User Study on Explainable Online Reinforcement Learning for Adaptive Systems](https://mlsanity.com/paper/2307.04098/) (2023)
- [Toward Affective XAI: Facial Affect Analysis for Understanding Explainable Human-AI Interactions](https://mlsanity.com/paper/2106.08761/) (2021)

## Abstract (arXiv)

Since Artificial Intelligence (AI) software uses techniques like deep lookahead search and stochastic optimization of huge neural networks to fit mammoth datasets, it often results in complex behavior that is difficult for people to understand. Yet organizations are deploying AI algorithms in many mission-critical settings. To trust their behavior, we must make AI intelligible, either by using inherently interpretable models or by developing new methods for explaining and controlling otherwise overwhelmingly complex decisions using local approximation, vocabulary alignment, and interactive explanation. This paper argues that intelligibility is essential, surveys recent work on building such systems, and highlights key directions for research.