# X-Pruner: eXplainable Pruning for Vision Transformers

arXiv: https://arxiv.org/abs/2303.04935 · HTML: https://mlsanity.com/paper/2303.04935/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 35 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 30 of 30)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](https://mlsanity.com/paper/2103.15679/) (2021)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [Chasing Sparsity in Vision Transformers: An End-to-End Exploration](https://mlsanity.com/paper/2106.04533/) (2021)
- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers](https://mlsanity.com/paper/2106.12620/) (2021)
- [SCOP: Scientific Control for Reliable Neural Network Pruning](https://mlsanity.com/paper/2010.10732/) (2020)
- [MiniViT: Compressing Vision Transformers with Weight Multiplexing](https://mlsanity.com/paper/2204.07154/) (2022)
- [Scalable Vision Transformers with Hierarchical Pooling](https://mlsanity.com/paper/2103.10619/) (2021)
- [Vision Transformer Pruning](https://mlsanity.com/paper/2104.08500/) (2021)
- [Unified Visual Transformer Compression](https://mlsanity.com/paper/2203.08243/) (2022)
- [PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance](https://mlsanity.com/paper/2206.12562/) (2022)
- [Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space](https://mlsanity.com/paper/2201.00814/) (2022)
- [Differentiable Subset Pruning of Transformer Heads](https://mlsanity.com/paper/2108.04657/) (2021)
- [SPDY: Accurate Pruning with Speedup Guarantees](https://mlsanity.com/paper/2201.13096/) (2022)
- [ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques](https://mlsanity.com/paper/2103.11367/) (2021)
- [Accelerating Neural Architecture Search via Proxy Data](https://mlsanity.com/paper/2106.04784/) (2021)

## Built on by (most cited first, 30 of 68)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Efficient Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2405.10739/) (2024)
- [MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer](https://mlsanity.com/paper/2403.02991/) (2024)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking](https://mlsanity.com/paper/2309.02031/) (2023)
- [AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer](https://mlsanity.com/paper/2407.12951/) (2024)
- [ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2406.09936/) (2024)
- [Isomorphic Pruning for Vision Models](https://mlsanity.com/paper/2407.04616/) (2024)
- [OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition](https://mlsanity.com/paper/2409.13652/) (2024)
- [ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models](https://mlsanity.com/paper/2310.02998/) (2023)
- [LPViT: Low-Power Semi-structured Pruning for Vision Transformers](https://mlsanity.com/paper/2407.02068/) (2024)
- [BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion](https://mlsanity.com/paper/2305.15798/) (2023)
- [From Efficient Multimodal Models to World Models: A Survey](https://mlsanity.com/paper/2407.00118/) (2024)
- [Accelerating Vision Transformers with Adaptive Patch Sizes](https://mlsanity.com/paper/2510.18091/) (2025)
- [Explainability of Vision Transformers: A Comprehensive Review and New Perspectives](https://mlsanity.com/paper/2311.06786/) (2023)
- [MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2401.14895/) (2024)
- [LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate](https://mlsanity.com/paper/2405.13985/) (2024)
- [Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformers](https://mlsanity.com/paper/2511.14751/) (2025)
- [UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer](https://mlsanity.com/paper/2401.06426/) (2024)
- [TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models](https://mlsanity.com/paper/2504.09897/) (2025)
- [TeleSparse: Practical Privacy-Preserving Verification of Deep Neural Networks](https://mlsanity.com/paper/2504.19274/) (2025)
- [Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP](https://mlsanity.com/paper/2502.18816/) (2025)
- [Data-independent Module-aware Pruning for Hierarchical Vision Transformers](https://mlsanity.com/paper/2404.13648/) (2024)
- [Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning](https://mlsanity.com/paper/2408.06798/) (2024)
- [Efficient Partitioning Vision Transformer on Edge Devices for Distributed Inference](https://mlsanity.com/paper/2410.11650/) (2024)
- [Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities](https://mlsanity.com/paper/2509.08302/) (2025)
- [HeSS: Head Sensitivity Score for Sparsity Redistribution in VGGT](https://mlsanity.com/paper/2603.25336/) (2026)
- [SNP: Structured Neuron-level Pruning to Preserve Attention Scores](https://mlsanity.com/paper/2404.11630/) (2024)
- [RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers](https://mlsanity.com/paper/2505.21847/) (2025)
- [FastFLUX: Pruning FLUX with Block-wise Replacement and Sandwich Training](https://mlsanity.com/paper/2506.10035/) (2025)

## Abstract (arXiv)

Recently vision transformer models have become prominent models for a range of tasks. These models, however, usually suffer from intensive computational costs and heavy memory requirements, making them impractical for deployment on edge platforms. Recent studies have proposed to prune transformers in an unexplainable manner, which overlook the relationship between internal units of the model and the target class, thereby leading to inferior performance. To alleviate this problem, we propose a novel explainable pruning framework dubbed X-Pruner, which is designed by considering the explainability of the pruning criterion. Specifically, to measure each prunable unit's contribution to predicting each target class, a novel explainability-aware mask is proposed and learned in an end-to-end manner. Then, to preserve the most informative units and learn the layer-wise pruning rate, we adaptively search the layer-wise threshold that differentiates between unpruned and pruned units based on their explainability-aware mask values. To verify and evaluate our method, we apply the X-Pruner on representative transformer models including the DeiT and Swin Transformer. Comprehensive simulation results demonstrate that the proposed X-Pruner outperforms the state-of-the-art black-box methods with significantly reduced computational costs and slight performance degradation.