# Explaining Convolutional Neural Networks using Softmax Gradient Layer-wise Relevance Propagation

arXiv: https://arxiv.org/abs/1908.04351 · HTML: https://mlsanity.com/paper/1908.04351/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2019.

## Built on by (most cited first, 29 of 29)

- [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](https://mlsanity.com/paper/2103.15679/) (2021)
- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [Transferable Adversarial Attack based on Integrated Gradients](https://mlsanity.com/paper/2205.13152/) (2022)
- [Gradient based Feature Attribution in Explainable AI: A Technical Review](https://mlsanity.com/paper/2403.10415/) (2024)
- [Towards Evaluating Explanations of Vision Transformers for Medical Imaging](https://mlsanity.com/paper/2304.06133/) (2023)
- [Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer](https://mlsanity.com/paper/2403.14552/) (2024)
- [Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs](https://mlsanity.com/paper/2503.16870/) (2025)
- [Explainable AI for Bioinformatics: Methods, Tools, and Applications](https://mlsanity.com/paper/2212.13261/) (2022)
- [GOAt: Explaining Graph Neural Networks via Graph Output Attribution](https://mlsanity.com/paper/2401.14578/) (2024)
- [Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP](https://mlsanity.com/paper/2502.18816/) (2025)
- [TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax](https://mlsanity.com/paper/2403.02920/) (2024)
- [Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation](https://mlsanity.com/paper/2602.20816/) (2026)
- [Point-to-Region Loss for Semi-Supervised Point-Based Crowd Counting](https://mlsanity.com/paper/2505.21943/) (2025)
- [Empowering CAM-Based Methods with Capability to Generate Fine-Grained and High-Faithfulness Explanations](https://mlsanity.com/paper/2303.09171/) (2023)
- [LibraGrad: Balancing Gradient Flow for Universally Better Vision Transformer Attributions](https://mlsanity.com/paper/2411.16760/) (2024)
- [PAMI: partition input and aggregate outputs for model interpretation](https://mlsanity.com/paper/2302.03318/) (2023)
- [Neural network interpretability with layer-wise relevance propagation: novel techniques for neuron selection and visualization](https://mlsanity.com/paper/2412.05686/) (2024)
- [ECGNet: A generative adversarial network (GAN) approach to the synthesis of 12-lead ECG signals from single lead inputs](https://mlsanity.com/paper/2310.03753/) (2023)
- [Towards Better Visualizing the Decision Basis of Networks via Unfold and Conquer Attribution Guidance](https://mlsanity.com/paper/2312.14201/) (2023)
- [Sparse Explanations of Neural Networks Using Pruned Layer-Wise Relevance Propagation](https://mlsanity.com/paper/2404.14271/) (2024)
- [FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks](https://mlsanity.com/paper/2408.02123/) (2024)
- [Layer-Wise Interpretation of Deep Neural Networks Using Identity Initialization](https://mlsanity.com/paper/2102.13333/) (2021)
- [Weakly Supervised Recovery of Semantic Attributes](https://mlsanity.com/paper/2103.11888/) (2021)
- [Interpretable Diabetic Retinopathy Diagnosis based on Biomarker Activation Map](https://mlsanity.com/paper/2212.06299/) (2022)
- [Enhancing Explainable AI: A Hybrid Approach Combining GradCAM and LRP for CNN Interpretability](https://mlsanity.com/paper/2405.12175/) (2024)
- [Matching-Based Few-Shot Semantic Segmentation Models Are Interpretable by Design](https://mlsanity.com/paper/2511.18163/) (2025)
- [Always Keep Your Promises: A Model-Agnostic Attribution Algorithm for Neural Networks](https://mlsanity.com/paper/2512.07010/) (2025)
- [The Neglected Baseline in Model Interpretation](https://mlsanity.com/paper/2605.22417/) (2026)
- [XtrAIn: Training-Guided Occlusion for Feature Attribution](https://mlsanity.com/paper/2606.10877/) (2026)

## Abstract (arXiv)

Convolutional Neural Networks (CNN) have become state-of-the-art in the field of image classification. However, not everything is understood about their inner representations. This paper tackles the interpretability and explainability of the predictions of CNNs for multi-class classification problems. Specifically, we propose a novel visualization method of pixel-wise input attribution called Softmax-Gradient Layer-wise Relevance Propagation (SGLRP). The proposed model is a class discriminate extension to Deep Taylor Decomposition (DTD) using the gradient of softmax to back propagate the relevance of the output probability to the input image. Through qualitative and quantitative analysis, we demonstrate that SGLRP can successfully localize and attribute the regions on input images which contribute to a target object's classification. We show that the proposed method excels at discriminating the target objects class from the other possible objects in the images. We confirm that SGLRP performs better than existing Layer-wise Relevance Propagation (LRP) based methods and can help in the understanding of the decision process of CNNs.