mlsanityData to 14 Oct 2026

Adaptive Token Sampling For Efficient Vision Transformers

arXiv 2111.15667 ↗First posted 30 Nov 2021cs.CVcited by 183

Not on the map. Why?

Cited by 63 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While state-of-the-art vision transformer models achieve promising results in image classification, they are computationally expensive and require many GFLOPs. Although the GFLOPs of a vision transformer can be decreased by reducing the number of tokens in the network, there is no setting that is optimal for all input images. In this work, we therefore introduce a differentiable parameter-free Adaptive Token Sampler (ATS) module, which can be plugged into any existing vision transformer architecture. ATS empowers vision transformers by scoring and adaptively sampling significant tokens. As a result, the number of tokens is not constant anymore and varies for each input image. By integrating ATS as an additional layer within the current transformer blocks, we can convert them into much more efficient vision transformers with an adaptive number of tokens. Since ATS is a parameter-free module, it can be added to the off-the-shelf pre-trained vision transformers as a plug and play module, thus reducing their GFLOPs without any additional training. Moreover, due to its differentiable design, one can also train a vision transformer equipped with ATS. We evaluate the efficiency of our module in both image and video classification tasks by adding it to multiple SOTA vision transformers. Our proposed module improves the SOTA by reducing their computational costs (GFLOPs) by 2X, while preserving their accuracy on the ImageNet, Kinetics-400, and Kinetics-600 datasets.

Key facts. Cited by 183 papers in the map (data through 14 Oct 2026). Builds on 62 of its 73 references that are in the map. First posted on arXiv 30 Nov 2021.

Builds on

62 of 73 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  11. 2017Non-local Neural Networkscited by 2,293
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
38 more
  1. 2017The Kinetics Human Action Video Datasetcited by 1,909
  2. 2018SlowFast Networks for Video Recognitioncited by 1,867
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  5. 2020Point Transformercited by 1,470
  6. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  8. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  9. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  10. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  11. 2020Designing Network Design Spacescited by 960
  12. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  13. 2021Multiscale Vision Transformerscited by 897
  14. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  15. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  16. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  17. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  18. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637
  19. 2021Transformer in Transformercited by 633
  20. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  21. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  22. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  23. 2018A Short Note about Kinetics-600cited by 333
  24. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  25. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  26. 2021PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformerscited by 286
  27. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  28. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  29. 2021Global Filter Networks for Image Classificationcited by 260
  30. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  31. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  32. 2020TEA: Temporal Excitation and Aggregation for Action Recognitioncited by 165
  33. 2019Adaptive Attention Span in Transformerscited by 149
  34. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  35. 2019STM: SpatioTemporal and Motion Encoding for Action Recognitioncited by 142
  36. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  37. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  38. 2019Star-Transformercited by 94

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 35; 2024: 46; 2025: 46; 2026: 50.

183 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  2. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  3. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 125
  4. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  5. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 81
  6. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 77
  7. 2023SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applicationscited by 66
  8. 2023Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Reviewcited by 62
  9. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 62
  10. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 60
  11. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  12. 2024Matryoshka Query Transformer for Large Vision-Language Modelscited by 50
38 more
  1. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  2. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  3. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  4. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 40
  5. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformerscited by 38
  6. 2024Attention-Driven Training-Free Efficiency Enhancement of Diffusion Modelscited by 38
  7. 2024Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identificationcited by 35
  8. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  9. 2025Similarity-Aware Token Pruning: Your VLM but Fastercited by 35
  10. 2023DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Modelscited by 31
  11. 2023Making Vision Transformers Efficient from A Token Sparsification Viewcited by 29
  12. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  13. 2024Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformerscited by 26
  14. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 26
  15. 2022Token Turing Machinescited by 25
  16. 2023Towards Video Anomaly Retrieval from Video Anomaly Detection: New Benchmarks and Modelcited by 25
  17. 2022Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networkscited by 24
  18. 2023Skip-Attention: Improving Vision Transformers by Paying Less Attentioncited by 24
  19. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  20. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 22
  21. 2024An Efficient and Effective Transformer Decoder-Based Framework for Multi-Task Visual Groundingcited by 21
  22. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformerscited by 21
  23. 2025TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Trainingcited by 21
  24. 2023Efficient Video Action Detection with Token Dropout and Context Refinementcited by 20
  25. 2023GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagationcited by 19
  26. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 19
  27. 2023MonoATT: Online Monocular 3D Object Detection with Adaptive Token Transformercited by 18
  28. 2024Hierarchical Graph Interaction Transformer with Dynamic Token Clustering for Camouflaged Object Detectioncited by 18
  29. 2025ToMA: Token Merge with Attention for Diffusion Modelscited by 18
  30. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17
  31. 2024On the Utility of 3D Hand Poses for Action Recognitioncited by 17
  32. 2025FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attentioncited by 17
  33. 2025Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learningcited by 16
  34. 2022EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokenscited by 15
  35. 2023Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimationcited by 15
  36. 2023Grounding-Prompter: Prompting LLM with Multimodal Information for Temporal Sentence Grounding in Long Videoscited by 14
  37. 2023Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformerscited by 13
  38. 2024Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Modelscited by 13

50 most cited of 183 All 183 (JSON)