mlsanityData to 7 Oct 2026

Token Merging: Your ViT But Faster

arXiv 2210.09461 ↗First posted 17 Oct 2022cs.CVcited by 995

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

We introduce Token Merging (ToMe), a simple method to increase the throughput of existing ViT models without needing to train. ToMe gradually combines similar tokens in a transformer using a general and light-weight matching algorithm that is as fast as pruning while being more accurate. Off-the-shelf, ToMe can 2x the throughput of state-of-the-art ViT-L @ 512 and ViT-H @ 518 models on images and 2.2x the throughput of ViT-L on video with only a 0.2-0.3% accuracy drop in each case. ToMe can also easily be applied during training, improving in practice training speed up to 2x for MAE fine-tuning on video. Training with ToMe further minimizes accuracy drop, leading to 2x the throughput of ViT-B on audio for only a 0.4% mAP drop. Qualitatively, we find that ToMe merges object parts into one token, even over multiple frames of video. Overall, ToMe's accuracy and speed are competitive with state-of-the-art on images, video, and audio.

Key facts. Cited by 995 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 49 of its 59 references that are in the map. First posted on arXiv 17 Oct 2022.

Builds on

49 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec
  2. 2021Learned Token Pruning for Transformers2 Jul

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
35 more
  1. 2017The Kinetics Human Action Video Datasetcited by 1,894
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  4. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  5. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  6. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  7. 2021Video Swin Transformercited by 1,090
  8. 2021Scaling Vision Transformerscited by 993
  9. 2021Multiscale Vision Transformerscited by 896
  10. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  11. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  12. 2019Are Sixteen Heads Really Better than One?cited by 803
  13. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  14. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  15. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  16. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  17. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  18. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  19. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  20. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  21. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  22. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  23. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  24. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  25. 2022Masked Autoencoders that Listencited by 271
  26. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 230
  27. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  28. 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
  29. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  30. 2021Token Pooling in Vision Transformerscited by 68
  31. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  32. 2021Less is More: Pay Less Attention in Vision Transformerscited by 53
  33. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  34. 2022CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Predictioncited by 39
  35. 2021A Study on Token Pruning for ColBERTcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 89; 2024: 193; 2025: 287; 2026: 424.

995 papers.

Sort:

By area:

  1. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  2. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  3. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  4. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  5. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  6. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  7. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  8. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  9. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  10. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  11. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  12. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
38 more
  1. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  2. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  3. 2023Token Merging for Fast Stable Diffusioncited by 220
  4. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  5. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  6. 2023Video Understanding with Large Language Models: A Surveycited by 203
  7. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  8. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  9. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  10. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  11. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  12. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  13. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  14. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 151
  15. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 149
  16. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  17. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  18. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  19. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 135
  20. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 133
  21. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  22. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  23. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 125
  24. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  25. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 117
  26. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  27. 2023VidToMe: Video Token Merging for Zero-Shot Video Editingcited by 111
  28. 2024DiT4Edit: Diffusion Transformer for Image Editingcited by 100
  29. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  30. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  31. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  32. 2025FastVGGT: Training-Free Acceleration of Visual Geometry Transformercited by 93
  33. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  34. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  35. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 88
  36. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  37. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 83
  38. 2024Streaming Dense Video Captioningcited by 82

50 most cited of 995 All 995 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Patch Ranking: Efficient CLIP by Learning to Rank Local Patches
  2. 2026MergeOver: Post-Training Token Merging for Recursive Vision Transformers
  3. 2025TrimTokenator-LC: Towards Adaptive Visual Token Pruning for Large Multimodal Models with Long Contexts
  4. 2024TPC-ViT: Token Propagation Controller for Efficient Vision Transformer
  5. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models
  6. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models
  7. 2024Token Pruning using a Lightweight Background Aware Vision Transformer
  8. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer