mlsanityData to 13 Oct 2026

Learning to Merge Tokens in Vision Transformers

arXiv 2202.12015 ↗First posted 24 Feb 2022cs.CV cs.LGcited by 64

Not on the map. Why?

Cited by 20 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers are widely applied to solve natural language understanding and computer vision tasks. While scaling up these architectures leads to improved performance, it often comes at the expense of much higher computational costs. In order for large-scale models to remain practical in real-world systems, there is a need for reducing their computational overhead. In this work, we present the PatchMerger, a simple module that reduces the number of patches or tokens the network has to process by merging them between two consecutive intermediate layers. We show that the PatchMerger achieves a significant speedup across various model sizes while matching the original performance both upstream and downstream after fine-tuning.

Key facts. Cited by 64 papers in the map (data through 13 Oct 2026). Builds on 15 of its 18 references that are in the map. First posted on arXiv 24 Feb 2022.

Builds on

15 of 18 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,539
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,961
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,094
  11. 2021Scaling Vision Transformerscited by 998
  12. 2021Scaling Vision with Sparse Mixture of Expertscited by 744
3 more
  1. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 625
  2. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  3. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 5; 2023: 10; 2024: 30; 2025: 13; 2026: 6.

64 papers.

Sort:

By area:

  1. 2023From Sparse to Soft Mixtures of Expertscited by 247
  2. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 241
  3. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  4. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  5. 2025FastVGGT: Training-Free Acceleration of Visual Geometry Transformercited by 95
  6. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  7. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
  8. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 59
  9. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 51
  10. 2024Don't Look Twice: Faster Video Transformers with Run-Length Tokenizationcited by 42
  11. 2023Vision Transformers with Mixed-Resolution Tokenizationcited by 35
  12. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
38 more
  1. 2022PatchDropout: Economizing Vision Transformers Using Patch Dropoutcited by 28
  2. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  3. 2024Exploring Token Pruning in Vision State Space Modelscited by 28
  4. 2023Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformerscited by 22
  5. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 21
  6. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformerscited by 21
  7. 2023MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformerscited by 19
  8. 2023Efficient Video Action Detection with Token Dropout and Context Refinementcited by 18
  9. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 18
  10. 2023Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformerscited by 16
  11. 2024LookupViT: Compressing visual information to a limited number of tokenscited by 16
  12. 2024Rethinking Token Reduction for State Space Modelscited by 15
  13. 2024SAT-HMR: Real-Time Multi-Person 3D Mesh Estimation via Scale-Adaptive Tokenscited by 15
  14. 2025AVGGT: Rethinking Global Attention for Accelerating VGGTcited by 14
  15. 2024Agglomerative Token Clusteringcited by 13
  16. 2024Stealing User Prompts from Mixture of Expertscited by 12
  17. 2024Conditional computation in neural networks: principles and research trendscited by 10
  18. 2024PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inferencecited by 10
  19. 2024Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edgecited by 8
  20. 2023Adaptive Computation Modules: Granular Conditional Computation For Efficient Inferencecited by 6
  21. 2024Learning to Rank Patches for Unbiased Image Redundancy Reductioncited by 6
  22. 2025Dynamic Vision Mambacited by 6
  23. 2025Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentationcited by 5
  24. 2024Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformercited by 4
  25. 2026Vista: Scene-Aware Optimization for Streaming Video Question Answering under Post-Hoc Queriescited by 4
  26. 2023Unlocking Fine-Grained Details with Wavelet-based High-Frequency Enhancement in Transformerscited by 3
  27. 2023Generalizable Imitation Learning Through Pre-Trained Representationscited by 3
  28. 2024Vision HgNN: An Electron-Micrograph is Worth Hypergraph of Hypernodescited by 3
  29. 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AIcited by 3
  30. 2026OmniRad: A Radiological Foundation Model for Multi-Task Medical Image Analysiscited by 3
  31. 2024A Manifold Representation of the Key in Vision Transformerscited by 2
  32. 2024Token Turing Machines are Efficient Vision Modelscited by 2
  33. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  34. 2024Parameter-Efficient Quantized Mixture-of-Experts Meets Vision-Language Instruction Tuning for Semiconductor Electron Micrograph Analysiscited by 1
  35. 2024Mr. DETR++: Instructive Multi-Route Training for Detection Transformers with Mixture-of-Expertscited by 1
  36. 2024Cross-Layer Cache Aggregation for Token Reduction in Ultra-Fine-Grained Image Recognitioncited by 1
  37. 2025Training-free Token Reduction for Vision Mambacited by 1
  38. 2025Foundry: Distilling 3D Foundation Models for the Edgecited by 1

50 most cited of 64 All 64 (JSON)