mlsanityData to 7 Oct 2026

HoliTom: Holistic Token Merging for Fast Video Large Language Models

arXiv 2505.21334 ↗First posted 27 May 2025cs.CVcited by 58

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Video large language models (video LLMs) excel at video comprehension but face significant computational inefficiency due to redundant video tokens. Existing token pruning methods offer solutions. However, approaches operating within the LLM (inner-LLM pruning), such as FastV, incur intrinsic computational overhead in shallow layers. In contrast, methods performing token pruning before the LLM (outer-LLM pruning) primarily address spatial redundancy within individual frames or limited temporal windows, neglecting the crucial global temporal dynamics and correlations across longer video sequences. This leads to sub-optimal spatio-temporal reduction and does not leverage video compressibility fully. Crucially, the synergistic potential and mutual influence of combining these strategies remain unexplored. To further reduce redundancy, we introduce HoliTom, a novel training-free holistic token merging framework. HoliTom employs outer-LLM pruning through global redundancy-aware temporal segmentation, followed by spatial-temporal merging to reduce visual tokens by over 90%, significantly alleviating the LLM's computational burden. Complementing this, we introduce a robust inner-LLM token similarity-based merging approach, designed for superior performance and compatibility with outer-LLM pruning. Evaluations demonstrate our method's promising efficiency-performance trade-off on LLaVA-OneVision-7B, reducing computational costs to 6.9% of FLOPs while maintaining 99.1% of the original performance. Furthermore, we achieve a 2.28x reduction in Time-To-First-Token (TTFT) and a 1.32x acceleration in decoding throughput, highlighting the practical benefits of our integrated pruning approach for efficient video LLMs inference.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 55 of its 72 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

55 of 72 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs23 Apr
  2. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model24 Mar
  3. 2025QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long Video Comprehension11 Mar
  4. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Models9 Jan
  5. 2024PruneVid: Visual Token Pruning for Efficient Video Large Language Models20 Dec
  6. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration26 Nov
  7. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models22 Nov
  8. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval2 Sep
  9. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  10. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 45

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
33 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  3. 2022Token Merging: Your ViT But Fastercited by 995
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  5. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  6. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  7. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  8. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  9. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  10. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  11. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  12. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  13. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  14. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  15. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  16. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  17. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  18. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  19. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  20. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  21. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  22. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  23. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 116
  24. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  25. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  26. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 63
  27. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  28. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 47
  29. 2024Don't Look Twice: Faster Video Transformers with Run-Length Tokenizationcited by 42
  30. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  31. 2024[CLS] Token Tells Everything Needed for Training-free Efficient MLLMscited by 26
  32. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  33. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 40.

58 papers.

Sort:

By area:

  1. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  2. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 34
  3. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Modelscited by 25
  4. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  5. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  6. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  7. 2025RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learningcited by 20
  8. 2025OBS-Diff: Accurate Pruning For Diffusion Models in One-Shotcited by 19
  9. 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Mergingcited by 14
  10. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  11. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10
  12. 2025StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understandingcited by 9
38 more
  1. 2026ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoningcited by 8
  2. 2025MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understandingcited by 7
  3. 2025UniComp: Rethinking Video Compression Through Informational Uniquenesscited by 7
  4. 2026DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compressioncited by 6
  5. 2026Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Modelscited by 5
  6. 2026EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMscited by 5
  7. 2025EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMscited by 4
  8. 2026ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modelingcited by 4
  9. 2026Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Samplingcited by 4
  10. 2026Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understandingcited by 4
  11. 2025Revisiting MLLM Token Technology through the Lens of Classical Visual Codingcited by 3
  12. 2026Video-Based Reward Modeling for Computer-Use Agentscited by 3
  13. 2026Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMscited by 3
  14. 2026G2TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Modelscited by 3
  15. 2026EarlyTom: Early Token Compression Completes Fast Video Understandingcited by 3
  16. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  17. 2026Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unificationcited by 2
  18. 2026V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Modelscited by 2
  19. 2026LDDR: Linear-DPP-Based Dynamic-Resolution Frame Sampling for Video MLLMscited by 2
  20. 2026OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Modelscited by 2
  21. 2026LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMscited by 2
  22. 2026Temporal Aware Pruning for Efficient Diffusion-based Video Generationcited by 2
  23. 2026Audio-Visual Exchange-Aware Token Pruning for Efficient Audio-Visual Captioningcited by 2
  24. 2026OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Modelscited by 2
  25. 2026WaveZip: Wavelet-Driven Space-Time Decoupling for Video Token Condensationcited by 2
  26. 2025MANI-Pure: Magnitude-Adaptive Noise Injection for Adversarial Purificationcited by 1
  27. 2026Efficient Scaling of LLM Training with Flexible Context Parallelismcited by 1
  28. 2026Dynamic Token Compression for Efficient Video Understanding through Reinforcement Learningcited by 1
  29. 2026EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMscited by 1
  30. 2026OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Modelscited by 1
  31. 2026Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruningcited by 1
  32. 2026OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Modelscited by 1
  33. 2026Allocation Before Ranking: Decoupled Token Compression for OmniLLMscited by 1
  34. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  35. 2025EventSTU: Event-Guided Efficient Spatio-Temporal Understanding for Video Large Language Modelscited by 0
  36. 2025ShaRP: SHAllow-LayeR Pruning for Efficient Video Large Language Modelscited by 0
  37. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategiescited by 0
  38. 2026Tango: Taming Visual Signals for Efficient Video Large Language Modelscited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models
  2. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs
  3. 2026GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Models
  4. 2025MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs
  5. 2026TTF: Temporal Token Fusion for Efficient Video-Language Model
  6. 2026Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models
  7. 2025AdaTP: Attention-Debiased Token Pruning for Video Large Language Models
  8. 2025Language-Guided Temporal Token Pruning for Efficient VideoLLM Processing