mlsanityData to 7 Oct 2026

DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification

arXiv 2106.02034 ↗First posted 3 Jun 2021cs.CV cs.AI cs.LGcited by 849

Not on the map. Why?

Cited by 186 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Attention is sparse in vision transformers. We observe the final prediction in vision transformers is only based on a subset of most informative tokens, which is sufficient for accurate image recognition. Based on this observation, we propose a dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input. Specifically, we devise a lightweight prediction module to estimate the importance score of each token given the current features. The module is added to different layers to prune redundant tokens hierarchically. To optimize the prediction module in an end-to-end manner, we propose an attention masking strategy to differentiably prune a token by blocking its interactions with other tokens. Benefiting from the nature of self-attention, the unstructured sparse tokens are still hardware friendly, which makes our framework easy to achieve actual speed-up. By hierarchically pruning 66% of the input tokens, our method greatly reduces 31%~37% FLOPs and improves the throughput by over 40% while the drop of accuracy is within 0.5% for various vision transformers. Equipped with the dynamic token sparsification framework, DynamicViT models can achieve very competitive complexity/accuracy trade-offs compared to state-of-the-art CNNs and vision transformers on ImageNet. Code is available at https://github.com/raoyongming/DynamicViT

Key facts. Cited by 849 papers in the map (data through 7 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

28 of 39 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  12. 2020Point Transformercited by 1,462
16 more
  1. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  3. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  4. 2020Designing Network Design Spacescited by 954
  5. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  6. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  7. 2021Transformer in Transformercited by 632
  8. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  9. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  10. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  11. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  12. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  13. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  14. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  15. 2019Star-Transformercited by 94
  16. 2020MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillationcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 81; 2023: 147; 2024: 144; 2025: 177; 2026: 270.

849 papers.

Sort:

By area:

  1. 2022Token Merging: Your ViT But Fastercited by 995
  2. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  3. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  4. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  5. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  6. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  7. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  8. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 328
  9. 2021PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformerscited by 284
  10. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  11. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
  12. 2021Global Filter Networks for Image Classificationcited by 257
38 more
  1. 2022Transformers in Medical Imaging: A Surveycited by 245
  2. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  3. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  4. 2022CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformerscited by 227
  5. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  6. 2023Token Merging for Fast Stable Diffusioncited by 220
  7. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  8. 2021Embracing Single Stride 3D Object Detector with Sparse Transformercited by 193
  9. 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 193
  10. 2022Focal Modulation Networkscited by 186
  11. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  12. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  13. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  14. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  15. 2022FlexiViT: One Model for All Patch Sizescited by 151
  16. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 148
  17. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  18. 2022Multimodal Token Fusion for Vision Transformerscited by 132
  19. 2021CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptationcited by 129
  20. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  21. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 125
  22. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  23. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  24. 2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
  25. 2024DiTFastAttn: Attention Compression for Diffusion Transformer Modelscited by 109
  26. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  27. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  28. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  29. 2022Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformercited by 99
  30. 2021Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsitycited by 98
  31. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  32. 2023Generalized Relation Modeling for Transformer Trackingcited by 93
  33. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  34. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  35. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  36. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  37. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  38. 2022Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformerscited by 81

50 most cited of 849 All 849 (JSON)