mlsanityData to 7 Oct 2026

Token Fusion: Bridging the Gap between Token Pruning and Token Merging

arXiv 2312.01026 ↗First posted 2 Dec 2023cs.CVcited by 123

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Vision Transformers (ViTs) have emerged as powerful backbones in computer vision, outperforming many traditional CNNs. However, their computational overhead, largely attributed to the self-attention mechanism, makes deployment on resource-constrained edge devices challenging. Multiple solutions rely on token pruning or token merging. In this paper, we introduce "Token Fusion" (ToFu), a method that amalgamates the benefits of both token pruning and token merging. Token pruning proves advantageous when the model exhibits sensitivity to input interpolations, while token merging is effective when the model manifests close to linear responses to inputs. We combine this to propose a new scheme called Token Fusion. Moreover, we tackle the limitations of average merging, which doesn't preserve the intrinsic feature norm, resulting in distributional shifts. To mitigate this, we introduce MLERP merging, a variant of the SLERP technique, tailored to merge multiple tokens while maintaining the norm distribution. ToFu is versatile, applicable to ViTs with or without additional training. Our empirical evaluations indicate that ToFu establishes new benchmarks in both classification and image generation tasks concerning computational efficiency and model accuracy.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 42 of its 52 references that are in the map. First posted on arXiv 2 Dec 2023.

Builds on

42 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Token Merging: Your ViT But Faster17 Oct
  2. 2021Learned Token Pruning for Transformers2 Jul

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
28 more
  1. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  2. 2021Multiscale Vision Transformerscited by 896
  3. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  4. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  5. 2019Are Sixteen Heads Really Better than One?cited by 803
  6. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  7. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  8. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  9. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  10. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  11. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  12. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  13. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  14. 2016Revisiting Batch Normalization For Practical Domain Adaptationcited by 285
  15. 2022Masked Autoencoders that Listencited by 271
  16. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  17. 2023Token Merging for Fast Stable Diffusioncited by 220
  18. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  19. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  20. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  21. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  22. 2021Token Pooling in Vision Transformerscited by 68
  23. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  24. 2020SMYRF: Efficient Attention using Asymmetric Clusteringcited by 43
  25. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  26. 2022CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Predictioncited by 39
  27. 2020Sub-Linear Memory: How to Make Performers SLiMcited by 18
  28. 2021A Study on Token Pruning for ColBERTcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 56; 2026: 50.

123 papers.

Sort:

By area:

  1. 2025From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeerscited by 124
  2. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  3. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  4. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  5. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  6. 2025Similarity-Aware Token Pruning: Your VLM but Fastercited by 32
  7. 2025SpeCa: Accelerating Diffusion Transformers with Speculative Feature Cachingcited by 27
  8. 2025FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximationcited by 23
  9. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 20
  10. 2025DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformerscited by 19
  11. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 18
  12. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17
38 more
  1. 2025Compute Only 16 Tokens in One Timestep: Accelerating Diffusion Transformers with Cluster-Driven Feature Cachingcited by 17
  2. 2025ToMA: Token Merge with Attention for Diffusion Modelscited by 17
  3. 2024Squat: Quant Small Language Models on the Edgecited by 16
  4. 2025Cached Adaptive Token Merging: Dynamic Token Reduction and Redundant Computation Elimination in Diffusion Modelcited by 16
  5. 2025FreqCa: Accelerating Diffusion Models via Frequency-Aware Cachingcited by 16
  6. 2025Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learningcited by 16
  7. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  8. 2025A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generationcited by 14
  9. 2024γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Modelscited by 13
  10. 2024Realistic Evaluation of Model Merging for Compositional Generalizationcited by 12
  11. 2025Token Pruning in Audio Transformers: Optimizing Performance and Decoding Patch Importancecited by 12
  12. 2024Conditional computation in neural networks: principles and research trendscited by 10
  13. 2026SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Modelscited by 10
  14. 2024PaPr: Training-Free One-Step Patch Pruning with Lightweight ConvNets for Faster Inferencecited by 9
  15. 2025SADA: Stability-guided Adaptive Diffusion Accelerationcited by 9
  16. 2025Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformerscited by 9
  17. 2025CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clusteringcited by 8
  18. 2025HERO: Hierarchical Extrapolation and Refresh for Efficient World Modelscited by 8
  19. 2025Let Features Decide Their Own Solvers: Hybrid Feature Caching for Diffusion Transformerscited by 8
  20. 2025HTTM: Head-wise Temporal Token Merging for Faster VGGTcited by 8
  21. 2024Efficient Time Series Processing for Transformers and State-Space Models through Token Mergingcited by 7
  22. 2024Efficient Visual Transformer by Learnable Token Mergingcited by 7
  23. 2025Forecasting When to Forecast: Accelerating Diffusion Models with Confidence-Gated Taylorcited by 6
  24. 2025Token Expand-Merge: Training-Free Token Compression for Vision-Language-Action Modelscited by 6
  25. 2026UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMscited by 6
  26. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  27. 2025Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessmentcited by 5
  28. 2025LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervisioncited by 5
  29. 2025RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layerscited by 5
  30. 2025One-Way Ticket:Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Modelscited by 5
  31. 2025Adaptive Token Merging for Efficient Transformer Semantic Communication at the Edgecited by 5
  32. 2025IPCV: Information-Preserving Compression for MLLM Visual Encoderscited by 5
  33. 2024Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformercited by 4
  34. 2025Lossless Token Merging Even Without Fine-Tuning in Vision Transformerscited by 4
  35. 2025Exploring The Visual Feature Space for Multimodal Neural Decodingcited by 4
  36. 2025AutoPrune: Each Complexity Deserves a Pruning Policycited by 4
  37. 2025Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applicationscited by 4
  38. 2025How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?cited by 4

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Multi-Scale And Token Mergence: Make Your ViT More Efficient
  2. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers
  3. 2023Learned Thresholds Token Merging and Pruning for Vision Transformers
  4. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  5. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers
  6. 2023Revisiting Token Pruning for Object Detection and Instance Segmentation
  7. 2024Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers
  8. 2023Training-Free Acceleration of ViTs with Delayed Spatial Merging