# Explainability Is in the Mind of the Beholder: Establishing the Foundations of Explainable Artificial Intelligence arXiv: https://arxiv.org/abs/2112.14466 ยท HTML: https://mlsanity.com/paper/2112.14466/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 51 of its 144 references that are in the map. - First posted on arXiv 29 Dec 2021. ## Builds on (most cited first, 30 of 51) - [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014) - [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017) - ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016) - [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017) - [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019) - [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017) - [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017) - [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017) - [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017) - [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018) - [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017) - [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016) - [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018) - [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018) - [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017) - [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019) - [FACE: Feasible and Actionable Counterfactual Explanations](https://mlsanity.com/paper/1909.09369/) (2019) - [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019) - [Visualizing Deep Neural Network Decisions: Prediction Difference Analysis](https://mlsanity.com/paper/1702.04595/) (2017) - [Explainable Machine Learning for Scientific Insights and Discoveries](https://mlsanity.com/paper/1905.08883/) (2019) - [Predictive Multiplicity in Classification](https://mlsanity.com/paper/1909.06677/) (2019) - [Explaining Explanations in AI](https://mlsanity.com/paper/1811.01439/) (2018) - [What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research](https://mlsanity.com/paper/2102.07817/) (2021) - [Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches](https://mlsanity.com/paper/1912.05100/) (2019) - [The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards](https://mlsanity.com/paper/1805.03677/) (2018) - [The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies](https://mlsanity.com/paper/2007.15911/) (2020) - [Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences](https://mlsanity.com/paper/1712.00547/) (2017) - [Explainable Artificial Intelligence: a Systematic Review](https://mlsanity.com/paper/2006.00093/) (2020) - [The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification](https://mlsanity.com/paper/1503.01161/) (2015) - [One Explanation Does Not Fit All: The Promise of Interactive Explanations for Machine Learning Transparency](https://mlsanity.com/paper/2001.09734/) (2020) ## Built on by (most cited first, 13 of 13) - [Explainable Predictive Maintenance: A Survey of Current Methods, Challenges and Opportunities](https://mlsanity.com/paper/2401.07871/) (2024) - [Is explainable AI a race against model complexity?](https://mlsanity.com/paper/2205.10119/) (2022) - [Mind the Gap! Bridging Explainable Artificial Intelligence and Human Understanding with Luhmann's Functional Theory of Communication](https://mlsanity.com/paper/2302.03460/) (2023) - [Cross-model Fairness: Empirical Study of Fairness and Ethics Under Model Multiplicity](https://mlsanity.com/paper/2203.07139/) (2022) - [(Un)reasonable Allure of Ante-hoc Interpretability for High-stakes Domains: Transparency Is Necessary but Insufficient for Comprehensibility](https://mlsanity.com/paper/2306.02312/) (2023) - [Interpretable and Explainable Machine Learning Methods for Predictive Process Monitoring: A Systematic Literature Review](https://mlsanity.com/paper/2312.17584/) (2023) - [What and How of Machine Learning Transparency: Building Bespoke Explainability Tools with Interoperable Algorithmic Components](https://mlsanity.com/paper/2209.03813/) (2022) - [Helpful, Misleading or Confusing: How Humans Perceive Fundamental Building Blocks of Artificial Intelligence Explanations](https://mlsanity.com/paper/2303.00934/) (2023) - [A Model for Intelligible Interaction Between Agents That Predict and Explain](https://mlsanity.com/paper/2301.01819/) (2023) - [Counterfactual Explanations for Clustering Models](https://mlsanity.com/paper/2409.12632/) (2024) - [Walking the Web of Concept-Class Relationships in Incrementally Trained Interpretable Models](https://mlsanity.com/paper/2502.20393/) (2025) - [Interpretable Neural System Dynamics: Combining Deep Learning with System Dynamics Modeling to Support Critical Applications](https://mlsanity.com/paper/2505.14428/) (2025) - [Towards explainable decision support using hybrid neural models for logistic terminal automation](https://mlsanity.com/paper/2509.07577/) (2025) ## Abstract (arXiv) Explainable artificial intelligence and interpretable machine learning are research domains growing in importance. Yet, the underlying concepts remain somewhat elusive and lack generally agreed definitions. While recent inspiration from social sciences has refocused the work on needs and expectations of human recipients, the field still misses a concrete conceptualisation. We take steps towards addressing this challenge by reviewing the philosophical and social foundations of human explainability, which we then translate into the technological realm. In particular, we scrutinise the notion of algorithmic black boxes and the spectrum of understanding determined by explanatory processes and explainees' background knowledge. This approach allows us to define explainability as (logical) reasoning applied to transparent insights (into, possibly black-box, predictive systems) interpreted under background knowledge and placed within a specific context -- a process that engenders understanding in a selected group of explainees. We then employ this conceptualisation to revisit strategies for evaluating explainability as well as the much disputed trade-off between transparency and predictive power, including its implications for ante-hoc and post-hoc techniques along with fairness and accountability established by explainability. We furthermore discuss components of the machine learning workflow that may be in need of interpretability, building on a range of ideas from human-centred explainability, with a particular focus on explainees, contrastive statements and explanatory processes. Our discussion reconciles and complements current research to help better navigate open questions -- rather than attempting to address any individual issue -- thus laying a solid foundation for a grounded discussion and future progress of explainable artificial intelligence and interpretable machine learning.