mlsanityData to 7 Oct 2026

ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming

arXiv 2412.20105 ↗First posted 28 Dec 2024cs.CVcited by 20

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) enhance their perceptual capabilities by integrating visual and textual information. However, processing the massive number of visual tokens incurs a significant computational cost. Existing analysis of the MLLM attention mechanisms remains shallow, leading to coarse-grain token pruning strategies that fail to effectively balance speed and accuracy. In this paper, we conduct a comprehensive investigation of MLLM attention mechanisms with LLaVA. We find that numerous visual tokens and partial attention computations are redundant during the decoding process. Based on this insight, we propose Spatial-Temporal Visual Token Trimming (ST3), a framework designed to accelerate MLLM inference without retraining. ST3 consists of two primary components: 1) Progressive Visual Token Pruning (PVTP), which eliminates inattentive visual tokens across layers, and 2) Visual Token Annealing (VTA), which dynamically reduces the number of visual tokens in each layer as the generated tokens grow. Together, these techniques deliver around 2× faster inference with only about 30% KV cache memory compared to the original LLaVA, while maintaining consistent performance across various datasets. Crucially, ST3 can be seamlessly integrated into existing pre-trained MLLMs, providing a plug-and-play solution for efficient inference.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 53 references that are in the map. First posted on arXiv 28 Dec 2024.

Builds on

34 of 53 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM2 Jul
  2. 2024Efficient Large Multi-modal Models via Visual Context Compression28 Jun
  3. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Models18 Jun
  4. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLM7 Jun
  5. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models31 May
  6. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference9 May
  7. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  8. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar
  9. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices28 Dec
  10. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers27 May

Other references · 24

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2022Token Merging: Your ViT But Fastercited by 995
12 more
  1. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  2. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  5. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  6. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  7. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  8. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  9. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  10. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformerscited by 37
  11. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  12. 2024Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Modelcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 11.

20 papers.

Sort:

By area:

  1. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 47
  2. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  3. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  4. 2025Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraintscited by 17
  5. 2025PropVG: End-to-End Proposal-Driven Visual Grounding with Multi-Granularity Discriminationcited by 10
  6. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  7. 2025METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understandingcited by 4
  8. 2026Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Modelscited by 4
  9. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 3
  10. 2026EarlyTom: Early Token Compression Completes Fast Video Understandingcited by 3
  11. 2025DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understandingcited by 2
  12. 2026Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unificationcited by 2
8 more
  1. 2026Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Modelscited by 2
  2. 2025Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Modelscited by 1
  3. 2026KiToke: Kernel-based Interval-aware Token Compression for Video Large Language Modelscited by 1
  4. 2026Q Cache: Visual Attention is Valuable in Less than Half of Decode Layers for Multimodal Large Language Modelcited by 0
  5. 2026HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Modelscited by 0
  6. 2026Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMscited by 0
  7. 2026ETC: Extreme Token Compression via Task-aware Visual Information Distillation in VLMscited by 0
  8. 2026Multi-Image Visual Token Pruning in Large Visual Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
  2. 2026Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction
  3. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See
  4. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
  5. 2026Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models
  6. 2025A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models
  7. 2026Fast-Slow Efficient Training for Multimodal Large Language Models via Visual Token Pruning
  8. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models