mlsanityData to 14 Oct 2026

Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality

arXiv 2505.18227 ↗First posted 23 May 2025cs.LG cs.AIcited by 50

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

In Transformer architectures, tokenstextemdash discrete units derived from raw datatextemdash are formed by segmenting inputs into fixed-length chunks. Each token is then mapped to an embedding, enabling parallel attention computations while preserving the input's essential information. Due to the quadratic computational complexity of transformer self-attention mechanisms, token reduction has primarily been used as an efficiency strategy. This is especially true in single vision and language domains, where it helps balance computational costs, memory usage, and inference latency. Despite these advances, this paper argues that token reduction should transcend its traditional efficiency-oriented role in the era of large generative models. Instead, we position it as a fundamental principle in generative modeling, critically influencing both model architecture and broader applications. Specifically, we contend that across vision, language, and multimodal systems, token reduction can: (i) facilitate deeper multimodal integration and alignment, (ii) mitigate "overthinking" and hallucinations, (iii) maintain coherence over long inputs, and (iv) enhance training stability, etc. We reframe token reduction as more than an efficiency measure. By doing so, we outline promising future directions, including algorithm design, reinforcement learning-guided token reduction, token optimization for in-context learning, agentic framework design, and broader ML and scientific domains.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 133 of its 189 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

133 of 189 references.

Closest ancestors · 25

Same direction, newest first

  1. 2025Streamline Without Sacrifice -- Squeeze out Computation Redundancy in LMM21 May
  2. 2025Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models20 May
  3. 2025Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration16 May
  4. 2025Saliency-driven Dynamic Token Pruning for Large Language Models6 Apr
  5. 2025MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization1 Apr
  6. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understanding24 Mar
  7. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model24 Mar
  8. 2025VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models21 Mar
  9. 2025Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Models20 Mar
  10. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMs6 Mar
  11. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models4 Mar
  12. 2025Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?17 Feb
13 more
  1. 2025Vision-centric Token Compression in Large Language Model2 Feb
  2. 2024PruneVid: Visual Token Pruning for Efficient Video Large Language Models20 Dec
  3. 2024iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models9 Dec
  4. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs2 Dec
  5. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification1 Dec
  6. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration26 Nov
  7. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models22 Nov
  8. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval2 Sep
  9. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  10. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar
  11. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer5 Mar
  12. 2023VidToMe: Video Token Merging for Zero-Shot Video Editing17 Dec
  13. 2023Token Merging for Fast Stable Diffusion30 Mar

Other references · 108

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2022Scalable Diffusion Models with Transformerscited by 6,054
  9. 2025Qwen2.5-VL Technical Reportcited by 5,587
  10. 2022Token Merging: Your ViT But Fastercited by 1,021
  11. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  12. 2021Transformer in Transformercited by 633
38 more
  1. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  2. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  3. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  4. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  5. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 384
  6. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  7. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  8. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  9. 2023Adapting Language Models to Compress Contextscited by 356
  10. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  11. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  12. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 279
  13. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 262
  14. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 249
  15. 2024Token-Budget-Aware LLM Reasoningcited by 247
  16. 2025Training Language Models to Reason Efficientlycited by 247
  17. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  18. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  19. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  20. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 192
  21. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  22. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 167
  23. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 158
  24. 2024Accelerating Diffusion Transformers with Token-wise Feature Cachingcited by 156
  25. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 141
  26. 2021Learned Token Pruning for Transformerscited by 139
  27. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 139
  28. 2021Patch Slimming for Efficient Vision Transformerscited by 133
  29. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 125
  30. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  31. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  32. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 97
  33. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 97
  34. 2024Matryoshka Multimodal Modelscited by 84
  35. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 81
  36. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Modelscited by 79
  37. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  38. 2022Efficient Transformers with Dynamic Token Poolingcited by 76

50 most cited of 108 All 108 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 34.

50 papers.

Sort:

By area:

  1. 2025Latent Collaboration in Multi-Agent Systemscited by 73
  2. 2025Ivy-Fake: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detectioncited by 34
  3. 2025RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memorycited by 27
  4. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  5. 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
  6. 2026Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokenscited by 22
  7. 2025AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisitioncited by 21
  8. 2025Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learningcited by 19
  9. 2026ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selectioncited by 18
  10. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 12
  11. 2025DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inferencecited by 11
  12. 2025Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Accelerationcited by 9
38 more
  1. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 8
  2. 2026Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradientscited by 8
  3. 2026Toward Native Multimodal Modeling: A Roadmapcited by 7
  4. 2025Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks?cited by 6
  5. 2025Restore-R1: Efficient Image Restoration Agents via Reinforcement Learning with Multimodal LLM Perceptual Feedbackcited by 6
  6. 2026AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognitioncited by 6
  7. 2026Token Economics for LLM Agents: A Dual-View Study from Computing and Economicscited by 6
  8. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 5
  9. 2025Less is More: Improving LLM Reasoning with Minimal Test-Time Interventioncited by 5
  10. 2025CARL: Criticality-Aware Agentic Reinforcement Learningcited by 5
  11. 2026Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Modelscited by 5
  12. 2025PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Modelscited by 4
  13. 2026ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modelingcited by 4
  14. 2026Three-Step Nav: A Hierarchical Global-Local Planner for Zero-Shot Vision-and-Language Navigationcited by 4
  15. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  16. 2026OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transportcited by 3
  17. 2026TokenPilot: Cache-Efficient Context Management for LLM Agentscited by 3
  18. 2026MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Modelscited by 2
  19. 2026Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understandingcited by 2
  20. 2025Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregationcited by 1
  21. 2026PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understandingcited by 1
  22. 2026ET-Prune: Evidence-Aware Dynamic Budgeting for Visual Token Pruning in Text-Rich MLLMscited by 1
  23. 2026PECKER: A Precisely Efficient Critical Knowledge Erasure Recipe For Machine Unlearning in Diffusion Modelscited by 0
  24. 2026CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Modelscited by 0
  25. 2026Why Training-Free Token Reduction Collapses: The Inherent Instability of Pairwise Scoring Signalscited by 0
  26. 2026Fre-Res: Frequency-Residual Video Token Compression for Efficient Video MLLMscited by 0
  27. 2026Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarizationcited by 0
  28. 2026HiLo-Token: Input-Adaptive High-Low Frequency Token Compression for Efficient Image Editingcited by 0
  29. 2026ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Modelscited by 0
  30. 20263DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answeringcited by 0
  31. 2026CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Modelscited by 0
  32. 2026Fine-Tuning Qwen3-27B for C-to-Rust Code Translation: A Three-Stage Curriculum of Pretraining, Debugging-Aware SFT, and Task-Specific SFTcited by 0
  33. 2026Capacity Overflow: A Blind Spot for Backdoor Attacks in Vision MoEcited by 0
  34. 2026Position Is All You Need: A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentationcited by 0
  35. 2026CoRe: Coherence and Relational Alignment for Multivariate Time Series Forecastingcited by 0
  36. 2026LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perceptioncited by 0
  37. 2026From Patching to Pruning Visual Computation in Vision Language Modelscited by 0
  38. 2026Understanding and Mitigating Token-Pruning-Induced Vulnerabilities in VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More
  2. 2026Do Vision Language Models Need to Process Image Tokens?
  3. 2026Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models
  4. 2026Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models
  5. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models
  6. 2025Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment
  7. 2025Dynamic Token Reduction during Generation for Vision Language Models
  8. 2026RUTA: Principled Visual Token Allocation via Rate-Utility Optimization