mlsanityData to 7 Oct 2026

Token Pooling in Vision Transformers

arXiv 2110.03860 ↗First posted 8 Oct 2021cs.CV cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

Despite the recent success in many applications, the high computational requirements of vision transformers limit their use in resource-constrained settings. While many existing methods improve the quadratic complexity of attention, in most vision transformers, self-attention is not the major computation bottleneck, e.g., more than 80% of the computation is spent on fully-connected layers. To improve the computational complexity of all layers, we propose a novel token downsampling method, called Token Pooling, efficiently exploiting redundancies in the images and intermediate token representations. We show that, under mild assumptions, softmax-attention acts as a high-dimensional low-pass (smoothing) filter. Thus, its output contains redundancy that can be pruned to achieve a better trade-off between the computational cost and accuracy. Our new technique accurately approximates a set of tokens by minimizing the reconstruction error caused by downsampling. We solve this optimization problem via cost-efficient clustering. We rigorously analyze and compare to prior downsampling methods. Our experiments show that Token Pooling significantly improves the cost-accuracy trade-off over the state-of-the-art downsampling. Token Pooling is a simple and effective operator that can benefit many architectures. Applied to DeiT, it achieves the same ImageNet top-1 accuracy using 42% fewer computations.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 42 of its 62 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

42 of 62 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
30 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  6. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  7. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  8. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  9. 2020Efficient Transformers: A Surveycited by 784
  10. 2018Image Transformercited by 770
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  12. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  13. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  14. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  15. 2019The State of Sparsity in Deep Neural Networkscited by 471
  16. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  17. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  18. 2021Random Feature Attentioncited by 350
  19. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  20. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  21. 2016ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentationcited by 291
  22. 2017ICNet for Real-Time Semantic Segmentation on High-Resolution Imagescited by 261
  23. 2020Sparse Sinkhorn Attentioncited by 250
  24. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  25. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  26. 2020Fast Transformers with Clustered Attentioncited by 123
  27. 2018Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networkscited by 56
  28. 2019Efficient Segmentation: Learning Downsampling Near Semantic Boundariescited by 33
  29. 2021Centroid Transformers: Learning to Abstract with Attentioncited by 20
  30. 2018Maintaining Natural Image Statistics with the Contextual Losscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 20; 2024: 18; 2025: 11; 2026: 7.

68 papers.

Sort:

By area:

  1. 2022Token Merging: Your ViT But Fastercited by 995
  2. 2023Token Merging for Fast Stable Diffusioncited by 220
  3. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  4. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  5. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  6. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  7. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  8. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78
  9. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  10. 2022CATs++: Boosting Cost Aggregation with Convolutions and Transformerscited by 53
  11. 2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Modelscited by 53
  12. 2022Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learningcited by 49
38 more
  1. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  2. 2024Don't Look Twice: Faster Video Transformers with Run-Length Tokenizationcited by 42
  3. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  4. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 39
  5. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformerscited by 37
  6. 2024Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identificationcited by 34
  7. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  8. 2022SimA: Simple Softmax-free Attention for Vision Transformerscited by 32
  9. 2024Token Pruning for Caching Better: 9 Times Acceleration on Stable Diffusion for Freecited by 30
  10. 2022Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learningcited by 27
  11. 2022Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networkscited by 23
  12. 2023Pointersect: Neural Rendering with Cloud-Ray Intersectioncited by 23
  13. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understandingcited by 22
  14. 2023GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagationcited by 19
  15. 2023Efficient Video Action Detection with Token Dropout and Context Refinementcited by 18
  16. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17
  17. 2022Learning Hierarchical Image Segmentation For Recognition and By Recognitioncited by 16
  18. 2025Cached Adaptive Token Merging: Dynamic Token Reduction and Redundant Computation Elimination in Diffusion Modelcited by 16
  19. 2023Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimationcited by 15
  20. 2025FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attentioncited by 15
  21. 2023Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformerscited by 13
  22. 2024Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Modelscited by 13
  23. 2023Make A Long Image Short: Adaptive Token Length for Vision Transformerscited by 10
  24. 2025DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMscited by 10
  25. 2022What Makes for Good Tokenizers in Vision Transformer?cited by 9
  26. 2023AutoFocusFormer: Image Segmentation off the Gridcited by 9
  27. 2024Leveraging Temporal Contextualization for Video Action Recognitioncited by 9
  28. 2024Efficient Time Series Processing for Transformers and State-Space Models through Token Mergingcited by 7
  29. 2024Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-trainingcited by 5
  30. 2025REN: Fast and Efficient Region Encodings from Patch-Based Image Encoderscited by 5
  31. 2025Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentationcited by 5
  32. 2025D'ej`a Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reusecited by 5
  33. 2023Training-Free Acceleration of ViTs with Delayed Spatial Mergingcited by 4
  34. 2023Efficient Vision Transformer for Human Pose Estimation via Patch Selectioncited by 4
  35. 2023SlowFormer: Universal Adversarial Patch for Attack on Compute and Energy Efficiency of Inference Efficient Vision Transformerscited by 3
  36. 2025AdaTP: Attention-Debiased Token Pruning for Video Large Language Modelscited by 3
  37. 2023PAUMER: Patch Pausing Transformer for Semantic Segmentationcited by 2
  38. 2024Semantic Prompting with Image-Token for Continual Learningcited by 2

50 most cited of 68 All 68 (JSON)