# Fundamental Limitation in Explaining AI

arXiv: https://arxiv.org/abs/2605.24727 · HTML: https://mlsanity.com/paper/2605.24727/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 23 May 2026.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [Algorithmic Recourse: from Counterfactual Explanations to Interventions](https://mlsanity.com/paper/2002.06278/) (2020)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNs](https://mlsanity.com/paper/2008.02312/) (2020)
- [Explaining Explanations: Axiomatic Feature Interactions for Deep Networks](https://mlsanity.com/paper/2002.04138/) (2020)
- [Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations](https://mlsanity.com/paper/2310.11207/) (2023)
- [Impossibility Theorems for Feature Attribution](https://mlsanity.com/paper/2212.11870/) (2022)
- [GLocalX -- From Local to Global Explanations of Black Box AI Models](https://mlsanity.com/paper/2101.07685/) (2021)
- [How Interpretable are Reasoning Explanations from Prompting Large Language Models?](https://mlsanity.com/paper/2402.11863/) (2024)
- [NormLime: A New Feature Importance Metric for Explaining Deep Neural Networks](https://mlsanity.com/paper/1909.04200/) (2019)
- [Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws](https://mlsanity.com/paper/2504.09597/) (2025)
- [In-context learning and Occam's razor](https://mlsanity.com/paper/2410.14086/) (2024)
- [A Theory of Interpretable Approximations](https://mlsanity.com/paper/2406.10529/) (2024)
- [Trade-off Between Efficiency and Consistency for Removal-based Explanations](https://mlsanity.com/paper/2210.17426/) (2022)
- [Conversational Complexity for Assessing Risk in Large Language Models](https://mlsanity.com/paper/2409.01247/) (2024)
- [Expected Grad-CAM: Towards gradient faithfulness](https://mlsanity.com/paper/2406.01274/) (2024)
- [GLEAMS: Bridging the Gap Between Local and Global Explanations](https://mlsanity.com/paper/2408.05060/) (2024)
- [Kolmogorov Complexity Bounds for LLM Steganography and a Perplexity-Based Detection Proxy](https://mlsanity.com/paper/2603.21567/) (2026)

## Abstract (arXiv)

While large-scale models such as LLMs and diffusion models have achieved practical success, public institutions have emphasized the importance of explainability in AI. Existing methods for explaining AI, however, are not designed to provide completely faithful explanations of the behavior of large-scale AI systems. Although a completely faithful and interpretable explanation of the behavior of an AI system might be useful for AI governance, it has not been known whether providing such an explanation is theoretically possible. In this paper, we mathematically prove a fundamental quadrilemma in explaining AI, stating that AI and its explanation cannot satisfy the following four conditions simultaneously: 1) the complexity of the operation environment, 2) the goodness of the AI's performance, 3) the interpretability of the AI's explanation, and 4) the complete faithfulness of the AI's explanation. This quadrilemma suggests that, in most applications where we cannot change the environment or sacrifice good AI performance and an interpretable explanation, we should give up complete faithfulness of explanations and should instead aim to explain only the parts that are important for applications. As a consequence, the quadrilemma implies that AI governance should be designed on the premise that the faithfulness of AI explanations is always incomplete.