mlsanityData to 7 Oct 2026

AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning

arXiv 2412.03248 ↗First posted 4 Dec 2024cs.CV cs.AI cs.CLcited by 43

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have enabled the creation of multi-modal LLMs that exhibit strong comprehension of visual data such as images and videos. However, these models usually rely on extensive visual tokens from visual encoders, leading to high computational demands, which limits their applicability in resource-constrained environments and for long-context tasks. In this work, we propose a training-free adaptive inference method for multi-modal LLMs that can accommodate a broad range of efficiency requirements with a minimum performance drop. Our method consists of a) iterative token merging based on embedding similarity before LLMs, and b) progressive token pruning within LLM layers based on multi-modal importance. With a minimalist design, our method can be applied to both video and image LLMs. Extensive experiments on diverse video and image benchmarks demonstrate that our method substantially reduces computation load (e.g., a 7-fold reduction in FLOPs) while preserving the performance of video and image LLMs. Further, at a similar computational cost, our method outperforms the state-of-the-art methods in long video understanding (e.g., +4.6 on MLVU). Additionally, our in-depth analysis provides insights into token redundancy and LLM layer behaviors, offering guidance for future research in designing efficient multi-modal LLMs. Our code is available at https://github.com/LaVi-Lab/AIM.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 82 of its 101 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

82 of 101 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction22 Oct
  2. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See8 Oct
  3. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
  4. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models16 Sep
  5. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference9 May
  6. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 75

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  13. 2022Token Merging: Your ViT But Fastercited by 995
  14. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  15. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  16. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  17. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  18. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  19. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  20. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  21. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  22. 2024Long Context Transfer from Language to Visioncited by 502
  23. 2021Dynamic Neural Networks: A Surveycited by 405
  24. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  25. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  26. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  27. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  28. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  29. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  30. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  31. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  32. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  33. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  34. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  35. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  36. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  37. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  38. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 22.

43 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  3. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  4. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  5. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  6. 2026VisionTrim: Unified Vision Token Compression for Training-Free MLLM Accelerationcited by 16
  7. 2025Learning to Inference Adaptively for Multimodal Large Language Modelscited by 12
  8. 2024iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Modelscited by 11
  9. 2025CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanismscited by 11
  10. 2025Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Coveringcited by 10
  11. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  12. 2025LoC-Path: Learning to Compress for Pathology Multimodal Large Language Modelscited by 6
31 more
  1. 2025METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understandingcited by 4
  2. 2025CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Modelscited by 4
  3. 2026Large Language Model as Token Compressor and Decompressorcited by 4
  4. 2026Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncationcited by 4
  5. 2025AdaTP: Attention-Debiased Token Pruning for Video Large Language Modelscited by 3
  6. 2025METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Modelscited by 3
  7. 2025xLLM Technical Reportcited by 3
  8. 2026Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoningcited by 3
  9. 2026FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answeringcited by 3
  10. 2025Streamline Without Sacrifice -- Squeeze out Computation Redundancy in LMMcited by 2
  11. 2025Video, How Do Your Tokens Merge?cited by 2
  12. 2025Chat with AI: The Surprising Turn of Real-time Video Communication from Human to AIcited by 2
  13. 2025Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Mergingcited by 2
  14. 2026SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypasscited by 2
  15. 2026Locality Matters for Training-Free Audio Token Compression in Audio-Language Modelscited by 2
  16. 2026Audio-Visual Exchange-Aware Token Pruning for Efficient Audio-Visual Captioningcited by 2
  17. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Modelscited by 2
  18. 2025Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understandingcited by 1
  19. 2025OmniSparse: Training-Aware Fine-Grained Sparse Attention for Long-Video MLLMscited by 1
  20. 2026PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understandingcited by 1
  21. 2026ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generationcited by 1
  22. 2026ConsensusDrop: Fusing Visual and Cross-Modal Saliency for Efficient Vision Language Modelscited by 0
  23. 2026ForensicZip: More Tokens are Better but Not Necessary in Forensic Vision-Language Modelscited by 0
  24. 2026Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMscited by 0
  25. 2026Networking-Aware Energy Efficiency in Agentic AI Inference: A Surveycited by 0
  26. 2026POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMscited by 0
  27. 2026VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Modelscited by 0
  28. 2026Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMscited by 0
  29. 2026CADER: Confidence-Aware Dynamic Evidence Reasoning for Long-Video Understandingcited by 0
  30. 2026Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMscited by 0
  31. 2026Beyond One-Size-Fits-All: Sample-Adaptive Strategy Routing for Vision Token Pruning in MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models
  2. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models
  3. 2025TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models
  4. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
  5. 2026SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Models
  6. 2026Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference
  7. 2026Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference
  8. 2025Token Sequence Compression for Efficient Multimodal Computing