mlsanityData to 14 Oct 2026

IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers

arXiv 2106.12620 ↗First posted 23 Jun 2021cs.CVcited by 127

Not on the map. Why?

Cited by 28 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The self-attention-based model, transformer, is recently becoming the leading backbone in the field of computer vision. In spite of the impressive success made by transformers in a variety of vision tasks, it still suffers from heavy computation and intensive memory costs. To address this limitation, this paper presents an Interpretability-Aware REDundancy REDuction framework (IA-RED2). We start by observing a large amount of redundant computation, mainly spent on uncorrelated input patches, and then introduce an interpretable module to dynamically and gracefully drop these redundant patches. This novel framework is then extended to a hierarchical structure, where uncorrelated tokens at different stages are gradually removed, resulting in a considerable shrinkage of computational cost. We include extensive experiments on both image and video tasks, where our method could deliver up to 1.4x speed-up for state-of-the-art models like DeiT and TimeSformer, by only sacrificing less than 0.7% accuracy. More importantly, contrary to other acceleration approaches, our method is inherently interpretable with substantial visual evidence, making vision transformer closer to a more human-understandable architecture while being lighter. We demonstrate that the interpretability that naturally emerged in our framework can outperform the raw attention learned by the original visual transformer, as well as those generated by off-the-shelf interpretation methods, with both qualitative and quantitative results. Project Page: http://people.csail.mit.edu/bpan/ia-red/.

Key facts. Cited by 127 papers in the map (data through 14 Oct 2026). Builds on 60 of its 70 references that are in the map. First posted on arXiv 23 Jun 2021.

Builds on

60 of 70 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  8. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
38 more
  1. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  7. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  9. 2020Point Transformercited by 1,470
  10. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  11. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  12. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  13. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  14. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  15. 2021Multiscale Vision Transformerscited by 897
  16. 2020Pre-Trained Image Processing Transformercited by 893
  17. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  18. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  19. 2021Going deeper with Image Transformerscited by 764
  20. 2020PCT: Point cloud transformercited by 732
  21. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  22. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  23. 2021Transformer in Transformercited by 633
  24. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  25. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  26. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  27. 2021Dynamic Neural Networks: A Surveycited by 408
  28. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 332
  29. 2020Sparse Sinkhorn Attentioncited by 251
  30. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  31. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  32. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  33. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 199
  34. 2019Full-Gradient Representation for Neural Network Visualizationcited by 191
  35. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 189
  36. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
  37. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  38. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 24; 2023: 32; 2024: 24; 2025: 27; 2026: 8.

127 papers.

Sort:

By area:

  1. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  2. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
  3. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 177
  4. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 167
  5. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 125
  6. 2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
  7. 2021Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsitycited by 98
  8. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  9. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 81
  10. 2022Unified Visual Transformer Compressioncited by 79
  11. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 68
  12. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
38 more
  1. 2022ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognitioncited by 63
  2. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  3. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 60
  4. 2021Efficient Video Transformers with Spatial-Temporal Token Selectioncited by 59
  5. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  6. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  7. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  8. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  9. 2024Scene Adaptive Sparse Transformer for Event-based Object Detectioncited by 39
  10. 2023Less is More: Focus Attention for Efficient DETRcited by 38
  11. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformerscited by 38
  12. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 37
  13. 2021Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attentioncited by 36
  14. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attentioncited by 35
  15. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  16. 2022Towards Efficient Adversarial Training on Vision Transformerscited by 30
  17. 2022Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Groundingcited by 29
  18. 2023STPrivacy: Spatio-Temporal Privacy-Preserving Action Recognitioncited by 29
  19. 2023Making Vision Transformers Efficient from A Token Sparsification Viewcited by 29
  20. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  21. 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28
  22. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Modelscited by 28
  23. 2024Exploring Token Pruning in Vision State Space Modelscited by 28
  24. 2021Self-slimmed Vision Transformercited by 27
  25. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  26. 2024Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformerscited by 26
  27. 2024Interpretable Image Classification with Adaptive Prototype-based Vision Transformerscited by 25
  28. 2021Pruning Self-attentions into Convolutional Layers in Single Pathcited by 24
  29. 2022Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networkscited by 24
  30. 2022Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Trainingcited by 24
  31. 2025FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximationcited by 24
  32. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformerscited by 21
  33. 2022CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Modelscited by 20
  34. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  35. 2023Training a Large Video Model on a Single Machine in a Daycited by 20
  36. 2021An Information Theory-inspired Strategy for Automatic Network Pruningcited by 17
  37. 2024MLP Can Be A Good Transformer Learnercited by 17
  38. 2022Auto-scaling Vision Transformers without Trainingcited by 15

50 most cited of 127 All 127 (JSON)