mlsanityData to 14 Oct 2026

AdaViT: Adaptive Vision Transformers for Efficient Image Recognition

arXiv 2111.15668 ↗First posted 30 Nov 2021cs.CVcited by 228

Not on the map. Why?

Cited by 48 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Built on top of self-attention mechanisms, vision transformers have demonstrated remarkable performance on a variety of vision tasks recently. While achieving excellent performance, they still require relatively intensive computational cost that scales up drastically as the numbers of patches, self-attention heads and transformer blocks increase. In this paper, we argue that due to the large variations among images, their need for modeling long-range dependencies between patches differ. To this end, we introduce AdaViT, an adaptive computation framework that learns to derive usage policies on which patches, self-attention heads and transformer blocks to use throughout the backbone on a per-input basis, aiming to improve inference efficiency of vision transformers with a minimal drop of accuracy for image recognition. Optimized jointly with a transformer backbone in an end-to-end manner, a light-weight decision network is attached to the backbone to produce decisions on-the-fly. Extensive experiments on ImageNet demonstrate that our method obtains more than 2x improvement on efficiency compared to state-of-the-art vision transformers with only 0.8% drop of accuracy, achieving good efficiency/accuracy trade-offs conditioned on different computational budgets. We further conduct quantitative and qualitative analysis on learned usage polices and provide more insights on the redundancy in vision transformers.

Key facts. Cited by 228 papers in the map (data through 14 Oct 2026). Builds on 47 of its 62 references that are in the map. First posted on arXiv 30 Nov 2021.

Builds on

47 of 62 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  10. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  11. 2019Searching for MobileNetV3cited by 1,994
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
35 more
  1. 2021Vision Transformers for Dense Predictioncited by 1,803
  2. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  3. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  4. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  5. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  6. 2021Video Swin Transformercited by 1,091
  7. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  8. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  9. 2021Multiscale Vision Transformerscited by 897
  10. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  11. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  12. 2021Transformer in Transformercited by 633
  13. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  14. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  15. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  16. 2021TransReID: Transformer-based Object Re-Identificationcited by 435
  17. 2021Early Convolutions Help Transformers See Bettercited by 396
  18. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 332
  19. 2021Voxel Transformer for 3D Object Detectioncited by 247
  20. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  21. 20203D Object Detection with Pointformercited by 194
  22. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 189
  23. 2017Convolutional Networks with Adaptive Inference Graphscited by 148
  24. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  25. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  26. 2016Spatially Adaptive Computation Time for Residual Networkscited by 118
  27. 2021VidTr: Video Transformer Without Convolutionscited by 112
  28. 2021Training Vision Transformers for Image Retrievalcited by 90
  29. 2020AR-Net: Adaptive Frame Resolution for Efficient Action Recognitioncited by 88
  30. 2019Improved Techniques for Training Adaptive Deep Networkscited by 66
  31. 2019LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognitioncited by 65
  32. 2018AutoFocus: Efficient Multi-Scale Inferencecited by 46
  33. 2020Learning When and Where to Zoom with Deep Reinforcement Learningcited by 43
  34. 2019Batch-Shaping for Learning Conditional Channel Gated Networkscited by 42
  35. 20202D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognitioncited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 16; 2023: 58; 2024: 58; 2025: 59; 2026: 36.

228 papers.

Sort:

By area:

  1. 2022Token Merging: Your ViT But Fastercited by 1,021
  2. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 241
  3. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  4. 2023Token Merging for Fast Stable Diffusioncited by 224
  5. 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 197
  6. 2022Focal Modulation Networkscited by 186
  7. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  8. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 125
  9. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 118
  10. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  11. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 81
  12. 2022ObjectFormer for Image Manipulation Detection and Localizationcited by 78
38 more
  1. 2024Dynamic Diffusion Transformercited by 62
  2. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 60
  3. 2022Prototypical Residual Networks for Anomaly Detection and Localizationcited by 56
  4. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 54
  5. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  6. 2023Scale-Aware Modulation Meet Transformercited by 49
  7. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  8. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  9. 2022TORE: Token Reduction for Efficient Human Mesh Recovery with Transformercited by 41
  10. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  11. 2024MaskGaussian: Adaptive 3D Gaussian Representation from Probabilistic Maskscited by 41
  12. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 40
  13. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  14. 2023Vision Transformers with Mixed-Resolution Tokenizationcited by 36
  15. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  16. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  17. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  18. 2022ResFormer: Scaling ViTs with Multi-Resolution Trainingcited by 34
  19. 2024Towards Language-Driven Video Inpainting via Multimodal Large Language Modelscited by 33
  20. 2023STPrivacy: Spatio-Temporal Privacy-Preserving Action Recognitioncited by 29
  21. 2023Making Vision Transformers Efficient from A Token Sparsification Viewcited by 29
  22. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 29
  23. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  24. 2022PPT: token-Pruned Pose Transformer for monocular and multi-view human pose estimationcited by 27
  25. 2023Latency-aware Unified Dynamic Networks for Efficient Image Recognitioncited by 26
  26. 2023Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generationcited by 26
  27. 2024Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformerscited by 26
  28. 2022Detection Hub: Unifying Object Detection Datasets via Query Adaptation on Language Embeddingcited by 25
  29. 2022Multi-Prompt Alignment for Multi-Source Unsupervised Domain Adaptationcited by 25
  30. 2024Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptationcited by 25
  31. 2022Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networkscited by 24
  32. 2023Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulationcited by 24
  33. 2025BlockDance: Reuse Structurally Similar Spatio-Temporal Features to Accelerate Diffusion Transformerscited by 24
  34. 2024Video Token Merging for Long-form Video Understandingcited by 23
  35. 2025FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputscited by 23
  36. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  37. 2023Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformerscited by 22
  38. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understandingcited by 22

50 most cited of 228 All 228 (JSON)