mlsanityData to 7 Oct 2026

AdaViT: Adaptive Tokens for Efficient Vision Transformer

arXiv 2112.07658 ↗First posted 14 Dec 2021cs.CV cs.LGcited by 360

Not on the map. Why?

Cited by 88 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce A-ViT, a method that adaptively adjusts the inference cost of vision transformer (ViT) for images of different complexity. A-ViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds. We reformulate Adaptive Computation Time (ACT) for this task, extending halting to discard redundant spatial tokens. The appealing architectural properties of vision transformers enables our adaptive token reduction mechanism to speed up inference without modifying the network architecture or inference hardware. We demonstrate that A-ViT requires no extra parameters or sub-network for halting, as we base the learning of adaptive halting on the original network parameters. We further introduce distributional prior regularization that stabilizes training compared to prior ACT approaches. On the image classification task (ImageNet1K), we show that our proposed A-ViT yields high efficacy in filtering informative spatial features and cutting down on the overall compute. The proposed method improves the throughput of DeiT-Tiny by 62% and DeiT-Small by 38% with only 0.3% accuracy drop, outperforming prior art by a large margin. Project page at https://a-vit.github.io/

Key facts. Cited by 360 papers in the map (data through 7 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 14 Dec 2021.

Builds on

44 of 58 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2018Neural Ordinary Differential Equationscited by 3,733
  9. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
32 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  5. 2017Mixed Precision Trainingcited by 1,135
  6. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  7. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  8. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  9. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  10. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  11. 2021Dynamic Neural Networks: A Surveycited by 405
  12. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  13. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  14. 2020Sparse Sinkhorn Attentioncited by 250
  15. 2017Distilling a Neural Network Into a Soft Decision Treecited by 230
  16. 2019Depth-Adaptive Transformercited by 227
  17. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  18. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  19. 2020Rethinking Transformer-based Set Prediction for Object Detectioncited by 156
  20. 2021PonderNet: Learning to Pondercited by 156
  21. 2017Convolutional Networks with Adaptive Inference Graphscited by 147
  22. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  23. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  24. 2021Generative Adversarial Transformerscited by 120
  25. 2016Spatially Adaptive Computation Time for Residual Networkscited by 118
  26. 2019Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inferencecited by 91
  27. 2021JFB: Jacobian-Free Backpropagation for Implicit Networkscited by 84
  28. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 69
  29. 2017Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Executioncited by 59
  30. 2020Fully Dynamic Inference with Deep Neural Networkscited by 19
  31. 2017Changing Model Behavior at Test-Time Using Reinforcement Learningcited by 15
  32. 2018IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classificationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 31; 2023: 77; 2024: 83; 2025: 78; 2026: 91.

360 papers.

Sort:

By area:

  1. 2022Token Merging: Your ViT But Fastercited by 995
  2. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  3. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  4. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  5. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 328
  6. 2022Confident Adaptive Language Modelingcited by 297
  7. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  8. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  9. 2023Token Merging for Fast Stable Diffusioncited by 220
  10. 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 193
  11. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  12. 2022Focal Modulation Networkscited by 186
38 more
  1. 2023EfficientAD: Accurate Visual Anomaly Detection at Millisecond-Level Latenciescited by 157
  2. 2022FlexiViT: One Model for All Patch Sizescited by 151
  3. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  4. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  5. 2022Global Context Vision Transformerscited by 98
  6. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  7. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  8. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  9. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78
  10. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  11. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  12. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  13. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 61
  14. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentationcited by 61
  15. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  16. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  17. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 59
  18. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  19. 2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Modelscited by 53
  20. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 53
  21. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
  22. 2022Green Hierarchical Vision Transformer for Masked Image Modelingcited by 50
  23. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  24. 2023Scale-Aware Modulation Meet Transformercited by 49
  25. 2024Matryoshka Query Transformer for Large Vision-Language Modelscited by 49
  26. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  27. 2022Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentationcited by 46
  28. 2025Infrared and Visible Image Fusion: From Data Compatibility to Task Adaptioncited by 45
  29. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  30. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  31. 2024Flextron: Many-in-One Flexible Large Language Modelcited by 42
  32. 2024Don't Look Twice: Faster Video Transformers with Run-Length Tokenizationcited by 42
  33. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  34. 2022Structural Pruning via Latency-Saliency Knapsackcited by 40
  35. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  36. 2024Adaptive Length Image Tokenization via Recurrent Allocationcited by 40
  37. 2024Scene Adaptive Sparse Transformer for Event-based Object Detectioncited by 39
  38. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 39

50 most cited of 360 All 360 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022SaiT: Sparse Vision Transformers through Adaptive Token Pruning
  2. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer
  3. 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI
  4. 2025ToFe: Lagged Token Freezing and Reusing for Efficient Vision Transformer Inference
  5. 2026ToaSt: Token Channel Selection and Structured Pruning for Efficient ViT
  6. 2026ViT-5: Vision Transformers for The Mid-2020s
  7. 2025TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers
  8. 2023No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling