# Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformers

arXiv: https://arxiv.org/abs/2408.12568 · HTML: https://mlsanity.com/paper/2408.12568/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 52 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [Explaining NonLinear Classification Decisions with Deep Taylor Decomposition](https://mlsanity.com/paper/1512.02479/) (2015)
- [EfficientFormer: Vision Transformers at MobileNet Speed](https://mlsanity.com/paper/2206.01191/) (2022)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond](https://mlsanity.com/paper/2202.06861/) (2022)
- [The Shattered Gradients Problem: If resnets are the answer, then what is the question?](https://mlsanity.com/paper/1702.08591/) (2017)
- [From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation](https://mlsanity.com/paper/2206.03208/) (2022)
- [Layer-adaptive sparsity for the Magnitude-based Pruning](https://mlsanity.com/paper/2010.07611/) (2020)
- [AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers](https://mlsanity.com/paper/2402.05602/) (2024)
- [A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation](https://mlsanity.com/paper/2306.07304/) (2023)

## Built on by (most cited first, 11 of 11)

- [Zero-shot Class Unlearning via Layer-wise Relevance Analysis and Neuronal Path Perturbation](https://mlsanity.com/paper/2410.23693/) (2024)
- [Attribution-Guided Pruning for Insight and Control: Circuit Discovery and Targeted Correction in Small-scale LLMs](https://mlsanity.com/paper/2506.13727/) (2025)
- [FedX: Explanation-Guided Pruning for Communication-Efficient Federated Learning in Remote Sensing](https://mlsanity.com/paper/2508.06256/) (2025)
- [MI-to-Mid Distilled Compression (M2M-DC): An Hybrid-Information-Guided-Block Pruning with Progressive Inner Slicing Approach to Model Compression](https://mlsanity.com/paper/2511.06842/) (2025)
- [Relevance-driven Input Dropout: an Explanation-guided Regularization Technique](https://mlsanity.com/paper/2505.21595/) (2025)
- [Interpretability-Aware Pruning for Efficient Medical Image Analysis](https://mlsanity.com/paper/2507.08330/) (2025)
- [C-SWAP: Explainability-Aware Structured Pruning for Efficient Neural Networks Compression](https://mlsanity.com/paper/2510.18636/) (2025)
- [An accuracy-aware extension to LRP-based pruning for CNNs to prevent cascading accuracy degradation in data-scarce transfer learning](https://mlsanity.com/paper/2511.10861/) (2025)
- [Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models](https://mlsanity.com/paper/2606.19350/) (2026)
- [Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs](https://mlsanity.com/paper/2606.19993/) (2026)
- [ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction](https://mlsanity.com/paper/2607.05356/) (2026)

## Abstract (arXiv)

To solve ever more complex problems, Deep Neural Networks are scaled to billions of parameters, leading to huge computational costs. An effective approach to reduce computational requirements and increase efficiency is to prune unnecessary components of these often over-parameterized networks. Previous work has shown that attribution methods from the field of eXplainable AI serve as effective means to extract and prune the least relevant network components in a few-shot fashion. We extend the current state by proposing to explicitly optimize hyperparameters of attribution methods for the task of pruning, and further include transformer-based networks in our analysis. Our approach yields higher model compression rates of large transformer- and convolutional architectures (VGG, ResNet, ViT) compared to previous works, while still attaining high performance on ImageNet classification tasks. Here, our experiments indicate that transformers have a higher degree of over-parameterization compared to convolutional neural networks. Code is available at https://github.com/erfanhatefi/Pruning-by-eXplaining-in-PyTorch.