# Leveraging Explanations in Interactive Machine Learning: An Overview

arXiv: https://arxiv.org/abs/2207.14526 · HTML: https://mlsanity.com/paper/2207.14526/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 121 of its 210 references that are in the map.
- First posted on arXiv 29 Jul 2022.

## Builds on (most cited first, 30 of 121)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations](https://mlsanity.com/paper/1703.03717/) (2017)

## Built on by (most cited first, 30 of 53)

- [Discover and Cure: Concept-aware Mitigation of Spurious Correlation](https://mlsanity.com/paper/2305.00650/) (2023)
- [Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts](https://mlsanity.com/paper/2305.19951/) (2023)
- [BEARS Make Neuro-Symbolic Models Aware of their Reasoning Shortcuts](https://mlsanity.com/paper/2402.12240/) (2024)
- [Interactive AI with Retrieval-Augmented Generation for Next Generation Networking](https://mlsanity.com/paper/2401.11391/) (2024)
- [Neuro-Symbolic Continual Learning: Knowledge, Reasoning Shortcuts and Concept Rehearsal](https://mlsanity.com/paper/2302.01242/) (2023)
- [Learning to Intervene on Concept Bottlenecks](https://mlsanity.com/paper/2308.13453/) (2023)
- [Interactive Explanations by Conflict Resolution via Argumentative Exchanges](https://mlsanity.com/paper/2303.15022/) (2023)
- [Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents](https://mlsanity.com/paper/2401.05821/) (2024)
- [Interpretability is in the Mind of the Beholder: A Causal Framework for Human-interpretable Representation Learning](https://mlsanity.com/paper/2309.07742/) (2023)
- [Learning by Self-Explaining](https://mlsanity.com/paper/2309.08395/) (2023)
- [Studying How to Efficiently and Effectively Guide Models with Explanations](https://mlsanity.com/paper/2303.11932/) (2023)
- [Reconsidering Faithfulness in Regular, Self-Explainable and Domain Invariant GNNs](https://mlsanity.com/paper/2406.15156/) (2024)
- [Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absent](https://mlsanity.com/paper/2502.11267/) (2025)
- [If Concept Bottlenecks are the Question, are Foundation Models the Answer?](https://mlsanity.com/paper/2504.19774/) (2025)
- [EXMOS: Explanatory Model Steering Through Multifaceted Explanations and Data Configurations](https://mlsanity.com/paper/2402.00491/) (2024)
- [Shortcuts and Identifiability in Concept-based Models from a Neuro-Symbolic Lens](https://mlsanity.com/paper/2502.11245/) (2025)
- [HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning](https://mlsanity.com/paper/2406.03997/) (2024)
- [Beyond Topological Self-Explainable GNNs: A Formal Explainability Perspective](https://mlsanity.com/paper/2502.02719/) (2025)
- [A survey on Concept-based Approaches For Model Improvement](https://mlsanity.com/paper/2403.14566/) (2024)
- [Unpacking Human-AI interactions: From interaction primitives to a design space](https://mlsanity.com/paper/2401.05115/) (2024)
- [Improving Human Verification of LLM Reasoning through Interactive Explanation Interfaces](https://mlsanity.com/paper/2510.22922/) (2025)
- [How Faithful are Self-Explainable GNNs?](https://mlsanity.com/paper/2308.15096/) (2023)
- [GNN Explanations that do not Explain and How to find Them](https://mlsanity.com/paper/2601.20815/) (2026)
- [Lessons Learned from EXMOS User Studies: A Technical Report Summarizing Key Takeaways from User Studies Conducted to Evaluate The EXMOS Platform](https://mlsanity.com/paper/2310.02063/) (2023)
- [The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Research](https://mlsanity.com/paper/2310.07882/) (2023)
- [Learning To Guide Human Decision Makers With Vision-Language Models](https://mlsanity.com/paper/2403.16501/) (2024)
- [An Explanatory Model Steering System for Collaboration between Domain Experts and AI](https://mlsanity.com/paper/2405.13038/) (2024)
- [Model free variable importance for high dimensional data](https://mlsanity.com/paper/2211.08414/) (2022)
- [One Explanation Does Not Fit XIL](https://mlsanity.com/paper/2304.07136/) (2023)
- [Explainable Interface for Human-Autonomy Teaming: A Survey](https://mlsanity.com/paper/2405.02583/) (2024)

## Abstract (arXiv)

Explanations have gained an increasing level of interest in the AI and Machine Learning (ML) communities in order to improve model transparency and allow users to form a mental model of a trained ML model. However, explanations can go beyond this one way communication as a mechanism to elicit user control, because once users understand, they can then provide feedback. The goal of this paper is to present an overview of research where explanations are combined with interactive capabilities as a mean to learn new models from scratch and to edit and debug existing ones. To this end, we draw a conceptual map of the state-of-the-art, grouping relevant approaches based on their intended purpose and on how they structure the interaction, highlighting similarities and differences between them. We also discuss open research issues and outline possible directions forward, with the hope of spurring further research on this blooming research topic.