mlsanityData to 14 Oct 2026

Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs

arXiv 2507.07990 ↗First posted 10 Jul 2025cs.CV cs.AIcited by 31

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Video large language models (LLMs) achieve strong video understanding by leveraging a large number of spatio-temporal tokens, but suffer from quadratic computational scaling with token count. To address this, we propose a training-free spatio-temporal token merging method, named STTM. Our key insight is to exploit local spatial and temporal redundancy in video data which has been overlooked in prior work. STTM first transforms each frame into multi-granular spatial tokens using a coarse-to-fine search over a quadtree structure, then performs directed pairwise merging across the temporal dimension. This decomposed merging approach outperforms existing token reduction methods across six video QA benchmarks. Notably, STTM achieves a 2× speed-up with only a 0.5% accuracy drop under a 50% token budget, and a 3× speed-up with just a 2% drop under a 30% budget. Moreover, STTM is query-agnostic, allowing KV cache reuse across different questions for the same video. The project page is available at https://www.jshyun.me/projects/sttm.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 48 of its 62 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

48 of 62 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Models30 Dec
  2. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models22 Nov
  3. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  4. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2024GPT-4o System Cardcited by 4,588
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
32 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  7. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  10. 2022Token Merging: Your ViT But Fastercited by 1,021
  11. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  12. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  13. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  14. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  15. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  16. 2022Efficiently Scaling Transformer Inferencecited by 539
  17. 2024Long Context Transfer from Language to Visioncited by 515
  18. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  19. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  20. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  21. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  22. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  23. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  24. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  25. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  26. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  27. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  28. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 147
  29. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 95
  30. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  31. 2024Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMscited by 37
  32. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 19.

31 papers.

Sort:

By area:

  1. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  2. 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Mergingcited by 18
  3. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 11
  4. 2025HTTM: Head-wise Temporal Token Merging for Faster VGGTcited by 9
  5. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  6. 2026VideoTemp-o3: Harmonizing Temporal Grounding and Video Understanding in Agentic Thinking-with-Videoscited by 7
  7. 2025FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understandingcited by 6
  8. 2025EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMscited by 6
  9. 2025D3ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMscited by 5
  10. 2026Stateful Token Reduction for Long-Video Hybrid VLMscited by 4
  11. 2026ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modelingcited by 4
  12. 2026TIR-Flow: Active Video Search and Reasoning with Frozen VLMscited by 3
19 more
  1. 2026Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMscited by 3
  2. 2026OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Modelscited by 3
  3. 2025Video Parallel Scaling: Aggregating Diverse Frame Subsets for VideoLLMscited by 2
  4. 2025Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Mergingcited by 2
  5. 2026DynaTok: Temporally Adaptive and Positional Bias-Aware Token Compression for Video-LLMscited by 2
  6. 2025ReGATE: Learning Faster and Better with Fewer Tokens in MLLMscited by 1
  7. 2025Foundry: Distilling 3D Foundation Models for the Edgecited by 1
  8. 2026Unified Spatio-Temporal Token Scoring for Efficient Video VLMscited by 1
  9. 2026OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Modelscited by 1
  10. 2025Sharp Eyes and Memory for VideoLLMs: Information-Aware Visual Token Pruning for Efficient and Reliable VideoLLM Reasoningcited by 0
  11. 2025ShaRP: SHAllow-LayeR Pruning for Efficient Video Large Language Modelscited by 0
  12. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategiescited by 0
  13. 2026Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMscited by 0
  14. 2026VLMaxxing through FrameMogging Training-Free Anti-Recomputation for Video Vision-Language Modelscited by 0
  15. 2026Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportationcited by 0
  16. 2026CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Modelscited by 0
  17. 2026CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understandingcited by 0
  18. 2026Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMscited by 0
  19. 2026FoveaTree: Information-Density Trees for Query-Guided Visual Token Pruning in Video Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention
  2. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Models
  3. 2026GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Models
  4. 2026KiToke: Kernel-based Interval-aware Token Compression for Video Large Language Models
  5. 2026Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models
  6. 2026Enhancing Visual Token Representations for Video Large Language Models via Training-Free Spatial-Temporal Pooling and Gridding
  7. 2025MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs
  8. 2026HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models