# Incremental XAI: Memorable Understanding of AI with Incremental Explanations

arXiv: https://arxiv.org/abs/2404.06733 · HTML: https://mlsanity.com/paper/2404.06733/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 75 references that are in the map.
- First posted on arXiv 10 Apr 2024.

## Builds on (most cited first, 24 of 24)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model](https://mlsanity.com/paper/1511.01644/) (2015)
- [Human-Centered Explainable AI (XAI): From Algorithms to User Experiences](https://mlsanity.com/paper/2110.10790/) (2021)
- [What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research](https://mlsanity.com/paper/2102.07817/) (2021)
- [To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making](https://mlsanity.com/paper/2102.09692/) (2021)
- [Supersparse Linear Integer Models for Optimized Medical Scoring Systems](https://mlsanity.com/paper/1502.04269/) (2015)
- [How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation](https://mlsanity.com/paper/1802.00682/) (2018)
- ["Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction](https://mlsanity.com/paper/2210.03735/) (2022)
- [GLocalX -- From Local to Global Explanations of Black Box AI Models](https://mlsanity.com/paper/2101.07685/) (2021)
- [Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory](https://mlsanity.com/paper/2205.05057/) (2022)
- [Why Does My Model Fail? Contrastive Local Explanations for Retail Forecasting](https://mlsanity.com/paper/1908.00085/) (2019)
- [Model Agnostic Multilevel Explanations](https://mlsanity.com/paper/2003.06005/) (2020)
- [Show or Suppress? Managing Input Uncertainty in Machine Learning Model Explanations](https://mlsanity.com/paper/2101.09498/) (2021)
- [Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation](https://mlsanity.com/paper/2109.10149/) (2021)
- [Brittle AI, Causal Confusion, and Bad Mental Models: Challenges and Successes in the XAI Program](https://mlsanity.com/paper/2106.05506/) (2021)
- [SLISEMAP: Supervised dimensionality reduction through local explanations](https://mlsanity.com/paper/2201.04455/) (2022)

## Built on by (most cited first, 8 of 8)

- [Privacy Risks and Preservation Methods in Explainable Artificial Intelligence: A Scoping Review](https://mlsanity.com/paper/2505.02828/) (2025)
- [DiagLink: A Dual-User Diagnostic Assistance System by Synergizing Experts with LLMs and Knowledge Graphs](https://mlsanity.com/paper/2601.20311/) (2026)
- [X-SYS: A Reference Architecture for Interactive Explanation Systems](https://mlsanity.com/paper/2602.12748/) (2026)
- [Transferable XAI: Relating Understanding Across Domains with Explanation Transfer](https://mlsanity.com/paper/2602.13675/) (2026)
- [Rules or Weights? Comparing User Understanding of Explainable AI Techniques with the Cognitive XAI-Adaptive Model](https://mlsanity.com/paper/2602.19620/) (2026)
- [CoAX: Cognitive-Oriented Attribution eXplanation User Model of Human Understanding of AI Explanations](https://mlsanity.com/paper/2604.27354/) (2026)
- [SketchXplain: Intuitive Visual Explanations of Image Classifiers with Sketches](https://mlsanity.com/paper/2606.17646/) (2026)
- [XAI-Arena: Can LLMs Assess the Quality of XAI Explanations?](https://mlsanity.com/paper/2609.09428/) (2026)

## Abstract (arXiv)

Many explainable AI (XAI) techniques strive for interpretability by providing concise salient information, such as sparse linear factors. However, users either only see inaccurate global explanations, or highly-varying local explanations. We propose to provide more detailed explanations by leveraging the human cognitive capacity to accumulate knowledge by incrementally receiving more details. Focusing on linear factor explanations (factors × values = outcome), we introduce Incremental XAI to automatically partition explanations for general and atypical instances by providing Base + Incremental factors to help users read and remember more faithful explanations. Memorability is improved by reusing base factors and reducing the number of factors shown in atypical cases. In modeling, formative, and summative user studies, we evaluated the faithfulness, memorability and understandability of Incremental XAI against baseline explanation methods. This work contributes towards more usable explanation that users can better ingrain to facilitate intuitive engagement with AI.