mlsanityData to 7 Oct 2026

Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs

arXiv 2409.10994 ↗First posted 17 Sep 2024cs.CL cs.AI cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

The rapid advancement of Multimodal Large Language Models (MLLMs) has led to remarkable performances across various domains. However, this progress is accompanied by a substantial surge in the resource consumption of these models. We address this pressing issue by introducing a new approach, Token Reduction using CLIP Metric (TRIM), aimed at improving the efficiency of MLLMs without sacrificing their performance. Inspired by human attention patterns in Visual Question Answering (VQA) tasks, TRIM presents a fresh perspective on the selection and reduction of image tokens. The TRIM method has been extensively tested across 12 datasets, and the results demonstrate a significant reduction in computational overhead while maintaining a consistent level of performance. This research marks a critical stride in efficient MLLM development, promoting greater accessibility and sustainability of high-performing models.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 62 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

38 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  2. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images18 Mar

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
24 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  4. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  5. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  6. 2024Yi: Open Foundation Models by 01.AIcited by 706
  7. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  8. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  9. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  10. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  11. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  12. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  13. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  14. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  15. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  16. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  17. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  18. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  19. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  20. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbonescited by 77
  21. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 76
  22. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteriacited by 27
  23. 2024Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Modelscited by 18
  24. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reductioncited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Improving Visual Reasoning with Iterative Evidence Refinementcited by 5
  2. 2026Rethinking Token Reduction for Large Vision-Language Modelscited by 2
  3. 2026Understanding and Mitigating Token-Pruning-Induced Vulnerabilities in VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Models
  2. 2025LFTR: Learning-Free Token Reduction for Multimodal Large Language Models
  3. 2026TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
  4. 2024freePruner: A Training-free Approach for Large Multimodal Model Acceleration
  5. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming
  6. 2026Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference
  7. 2024Efficient Large Multi-modal Models via Visual Context Compression
  8. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models