# Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvement

arXiv: https://arxiv.org/abs/2203.08008 · HTML: https://mlsanity.com/paper/2203.08008/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 73 of its 134 references that are in the map.
- First posted on arXiv 15 Mar 2022.

## Builds on (most cited first, 30 of 73)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering](https://mlsanity.com/paper/1811.03728/) (2018)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers](https://mlsanity.com/paper/1811.04918/) (2018)
- [Few-Shot Learning with Graph Neural Networks](https://mlsanity.com/paper/1711.04043/) (2017)
- [Top-down Neural Attention by Excitation Backprop](https://mlsanity.com/paper/1608.00507/) (2016)

## Built on by (most cited first, 30 of 48)

- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [Position: Explain to Question not to Justify](https://mlsanity.com/paper/2402.13914/) (2024)
- [Reveal to Revise: An Explainable AI Life Cycle for Iterative Bias Correction of Deep Models](https://mlsanity.com/paper/2303.12641/) (2023)
- [Establishing and Evaluating Trustworthy AI: Overview and Research Challenges](https://mlsanity.com/paper/2411.09973/) (2024)
- [Optimizing Explanations by Network Canonization and Hyperparameter Search](https://mlsanity.com/paper/2211.17174/) (2022)
- [Explainable AI for Bioinformatics: Methods, Tools, and Applications](https://mlsanity.com/paper/2212.13261/) (2022)
- [A Close Look at Decomposition-based XAI-Methods for Transformer Language Models](https://mlsanity.com/paper/2502.15886/) (2025)
- [Concept Distillation: Leveraging Human-Centered Explanations for Model Improvement](https://mlsanity.com/paper/2311.15303/) (2023)
- [A survey on Concept-based Approaches For Model Improvement](https://mlsanity.com/paper/2403.14566/) (2024)
- [XEdgeAI: A Human-centered Industrial Inspection Framework with Data-centric Explainable Edge AI Approach](https://mlsanity.com/paper/2407.11771/) (2024)
- [Strategies to exploit XAI to improve classification systems](https://mlsanity.com/paper/2306.05801/) (2023)
- [Towards Desiderata-Driven Design of Visual Counterfactual Explainers](https://mlsanity.com/paper/2506.14698/) (2025)
- [Controllable Continual Test-Time Adaptation](https://mlsanity.com/paper/2405.14602/) (2024)
- [Explainable artificial intelligence approaches for brain-computer interfaces: a review and design space](https://mlsanity.com/paper/2312.13033/) (2023)
- [The FIX Benchmark: Extracting Features Interpretable to eXperts](https://mlsanity.com/paper/2409.13684/) (2024)
- [Explainable Artificial Intelligence (XAI) for Malware Analysis: A Survey of Techniques, Applications, and Open Challenges](https://mlsanity.com/paper/2409.13723/) (2024)
- [LibraGrad: Balancing Gradient Flow for Universally Better Vision Transformer Attributions](https://mlsanity.com/paper/2411.16760/) (2024)
- [Cross- and Intra-image Prototypical Learning for Multi-label Disease Diagnosis and Interpretation](https://mlsanity.com/paper/2411.04607/) (2024)
- [Reliability Scores from Saliency Map Clusters for Improved Image-based Harvest-Readiness Prediction in Cauliflower](https://mlsanity.com/paper/2305.15149/) (2023)
- [Iterative Graph Neural Network Enhancement via Frequent Subgraph Mining of Explanations](https://mlsanity.com/paper/2403.07849/) (2024)
- [X-SHIELD: Regularization for eXplainable Artificial Intelligence](https://mlsanity.com/paper/2404.02611/) (2024)
- [xAI-Drop: Don't Use What You Cannot Explain](https://mlsanity.com/paper/2407.20067/) (2024)
- [Beyond explaining: XAI-based Adaptive Learning with SHAP Clustering for Energy Consumption Prediction](https://mlsanity.com/paper/2402.04982/) (2024)
- [Explainable Interface for Human-Autonomy Teaming: A Survey](https://mlsanity.com/paper/2405.02583/) (2024)
- [Locally Testing Model Detections for Semantic Global Concepts](https://mlsanity.com/paper/2405.17523/) (2024)
- [A Survey on Interpretability in Visual Recognition](https://mlsanity.com/paper/2507.11099/) (2025)
- [XAI-Enhanced Semantic Segmentation Models for Visual Quality Inspection](https://mlsanity.com/paper/2401.09900/) (2024)
- [Quantifying Spatial Domain Explanations in BCI using Earth Mover's Distance](https://mlsanity.com/paper/2405.01277/) (2024)
- [Learning Visually Grounded Domain Ontologies via Embodied Conversation and Explanation](https://mlsanity.com/paper/2412.09770/) (2024)

## Abstract (arXiv)

Explainable Artificial Intelligence (XAI) is an emerging research field bringing transparency to highly complex and opaque machine learning (ML) models. Despite the development of a multitude of methods to explain the decisions of black-box classifiers in recent years, these tools are seldomly used beyond visualization purposes. Only recently, researchers have started to employ explanations in practice to actually improve models. This paper offers a comprehensive overview over techniques that apply XAI practically for improving various properties of ML models, and systematically categorizes these approaches, comparing their respective strengths and weaknesses. We provide a theoretical perspective on these methods, and show empirically through experiments on toy and realistic settings how explanations can help improve properties such as model generalization ability or reasoning, among others. We further discuss potential caveats and drawbacks of these methods. We conclude that while model improvement based on XAI can have significant beneficial effects even on complex and not easily quantifyable model properties, these methods need to be applied carefully, since their success can vary depending on a multitude of factors, such as the model and dataset used, or the employed explanation method.