# Gradient based Feature Attribution in Explainable AI: A Technical Review

arXiv: https://arxiv.org/abs/2403.10415 · HTML: https://mlsanity.com/paper/2403.10415/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 102 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 30 of 64)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [Top-down Neural Attention by Excitation Backprop](https://mlsanity.com/paper/1608.00507/) (2016)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)

## Built on by (most cited first, 30 of 34)

- [Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulness](https://mlsanity.com/paper/2405.18915/) (2024)
- [Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Models](https://mlsanity.com/paper/2504.00573/) (2025)
- [TracLLM: A Generic Framework for Attributing Long Context LLMs](https://mlsanity.com/paper/2506.04202/) (2025)
- [Explainable Artificial Intelligence for Medical Applications: A Review](https://mlsanity.com/paper/2412.01829/) (2024)
- [Adaptive GoGI-Skip: Coupling Goal-Gradient Importance with Dynamic Uncertainty for Efficient Reasoning](https://mlsanity.com/paper/2505.08392/) (2025)
- [Probing Deep into Temporal Profile Makes the Infrared Small Target Detector Much Better](https://mlsanity.com/paper/2506.12766/) (2025)
- [The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution](https://mlsanity.com/paper/2601.15075/) (2026)
- [AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruption](https://mlsanity.com/paper/2508.03793/) (2025)
- [GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs](https://mlsanity.com/paper/2507.18043/) (2025)
- [EDformer: Embedded Decomposition Transformer for Interpretable Multivariate Time Series Predictions](https://mlsanity.com/paper/2412.12227/) (2024)
- [Towards Unified Attribution in Explainable AI, Data-Centric AI, and Mechanistic Interpretability](https://mlsanity.com/paper/2501.18887/) (2025)
- [Safety Must Precede the Deployment of Open-Ended AI](https://mlsanity.com/paper/2502.04512/) (2025)
- [Response Uncertainty and Probe Modeling: Two Sides of the Same Coin in LLM Interpretability?](https://mlsanity.com/paper/2505.18575/) (2025)
- [GIFT: A Framework Towards Global Interpretable Faithful Textual Explanations of Vision Classifiers](https://mlsanity.com/paper/2411.15605/) (2024)
- [Measuring the (Un)Faithfulness of Concept-Based Explanations](https://mlsanity.com/paper/2504.10833/) (2025)
- [Enhancing Feature Selection and Interpretability in AI Regression Tasks Through Feature Attribution](https://mlsanity.com/paper/2409.16787/) (2024)
- [Dynamic Information Sub-Selection for Decision Support](https://mlsanity.com/paper/2410.23423/) (2024)
- [Conversational Explanations: Discussing Explainable AI with Non-AI Experts](https://mlsanity.com/paper/2503.16444/) (2025)
- [Explainable but Vulnerable: Adversarial Attacks on XAI Explanation in Cybersecurity Applications](https://mlsanity.com/paper/2510.03623/) (2025)
- [When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs](https://mlsanity.com/paper/2510.09106/) (2025)
- [Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction](https://mlsanity.com/paper/2412.13110/) (2024)
- [From Abstract to Actionable: Pairwise Shapley Values for Explainable AI](https://mlsanity.com/paper/2502.12525/) (2025)
- [An Explainable Neural Radiomic Sequence Model with Spatiotemporal Continuity for Quantifying 4DCT-based Pulmonary Ventilation](https://mlsanity.com/paper/2503.23898/) (2025)
- [An Intelligent and Privacy-Preserving Digital Twin Model for Aging-in-Place](https://mlsanity.com/paper/2504.03798/) (2025)
- [Direct Preference Optimization for Adaptive Concept-based Explanations](https://mlsanity.com/paper/2505.15626/) (2025)
- [Building Transparency in Deep Learning-Powered Network Traffic Classification: A Traffic-Explainer Framework](https://mlsanity.com/paper/2509.18007/) (2025)
- [Explainable Graph Neural Networks: Understanding Brain Connectivity and Biomarkers in Dementia](https://mlsanity.com/paper/2509.18568/) (2025)
- [Onto-Epistemological Analysis of AI Explanations](https://mlsanity.com/paper/2510.02996/) (2025)
- [BAED: a New Paradigm for Few-shot Graph Learning with Explanation in the Loop](https://mlsanity.com/paper/2603.01941/) (2026)
- [Explanation of Dynamic Physical Field Predictions using WassersteinGrad: Application to Autoregressive Weather Forecasting](https://mlsanity.com/paper/2604.22580/) (2026)

## Abstract (arXiv)

The surge in black-box AI models has prompted the need to explain the internal mechanism and justify their reliability, especially in high-stakes applications, such as healthcare and autonomous driving. Due to the lack of a rigorous definition of explainable AI (XAI), a plethora of research related to explainability, interpretability, and transparency has been developed to explain and analyze the model from various perspectives. Consequently, with an exhaustive list of papers, it becomes challenging to have a comprehensive overview of XAI research from all aspects. Considering the popularity of neural networks in AI research, we narrow our focus to a specific area of XAI research: gradient based explanations, which can be directly adopted for neural network models. In this review, we systematically explore gradient based explanation methods to date and introduce a novel taxonomy to categorize them into four distinct classes. Then, we present the essence of technique details in chronological order and underscore the evolution of algorithms. Next, we introduce both human and quantitative evaluations to measure algorithm performance. More importantly, we demonstrate the general challenges in XAI and specific challenges in gradient based explanations. We hope that this survey can help researchers understand state-of-the-art progress and their corresponding disadvantages, which could spark their interest in addressing these issues in future work.