mlsanityData to 7 Oct 2026

Efficient Large Multi-modal Models via Visual Context Compression

arXiv 2406.20092 ↗First posted 28 Jun 2024cs.CVcited by 46

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

While significant advancements have been made in compressed representations for text embeddings in large language models (LLMs), the compression of visual tokens in multi-modal LLMs (MLLMs) has remained a largely overlooked area. In this work, we present the study on the analysis of redundancy concerning visual tokens and efficient training within these models. Our initial experiments show that eliminating up to 70% of visual tokens at the testing stage by simply average pooling only leads to a minimal 3% reduction in visual question answering accuracy on the GQA benchmark, indicating significant redundancy in visual context. Addressing this, we introduce Visual Context Compressor, which reduces the number of visual tokens to enhance training and inference efficiency without sacrificing performance. To minimize information loss caused by the compression on visual tokens while maintaining training efficiency, we develop LLaVolta as a light and staged training scheme that incorporates stage-wise visual context compression to progressively compress the visual tokens from heavily to lightly compression during training, yielding no loss of information when testing. Extensive experiments demonstrate that our approach enhances the performance of MLLMs in both image-language and video-language understanding, while also significantly cutting training costs and improving inference efficiency.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 52 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

37 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
24 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  9. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  11. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  12. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  13. 2022Exploring Length Generalization in Large Language Modelscited by 204
  14. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  15. 2021TransFG: A Transformer Architecture for Fine-grained Recognitioncited by 131
  16. 2021TransMix: Attend to Mix for Vision Transformerscited by 79
  17. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  18. 2022Staged Training for Transformer Language Modelscited by 49
  19. 2022Token Dropping for Efficient BERT Pretrainingcited by 42
  20. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  21. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  22. 2022Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selectioncited by 13
  23. 2024AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogiescited by 13
  24. 2023Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokenscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 19; 2026: 22.

46 papers.

Sort:

By area:

  1. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  2. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  3. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  4. 2024Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Accelerationcited by 40
  5. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  6. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  7. 2025Eyes Wide Open: Ego Proactive Video-LLM for Streaming Videocited by 25
  8. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimmingcited by 20
  9. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimizationcited by 20
  10. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Modelscited by 19
  11. 2025A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Modelscited by 18
  12. 2025Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Modelscited by 15
34 more
  1. 2025Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methodscited by 15
  2. 2026HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exitcited by 15
  3. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 8
  4. 2025MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understandingcited by 7
  5. 2025Delta-LLaVA: Base-then-Specialize Alignment for Token-Efficient Vision-Language Modelscited by 7
  6. 2026Thinking with Spatial Code for Physical-World Video Reasoningcited by 7
  7. 2026Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decodingcited by 6
  8. 2026What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Modelscited by 5
  9. 2025CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Modelscited by 4
  10. 2026Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Modelscited by 4
  11. 2025VCM: Vision Concept Modeling Based on Implicit Contrastive Learning with Vision-Language Instruction Fine-Tuningcited by 3
  12. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 3
  13. 2025EEG-VLM: A Hierarchical Vision-Language Model with Multi-Level Feature Alignment and Visually Enhanced Language-Guided Reasoning for EEG Image-Based Sleep Stage Predictioncited by 3
  14. 2026Where to Focus: Query-Modulated Multimodal Keyframe Selection for Long Video Understandingcited by 3
  15. 2026Sub-Token Routing for KV Cache Compressioncited by 3
  16. 2025Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspectivecited by 2
  17. 2025GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Modelscited by 2
  18. 2026Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMscited by 2
  19. 2026Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Modelscited by 2
  20. 2026Visual Token Compression Enhances Robustness of MLLMscited by 2
  21. 2025Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Modelscited by 1
  22. 2026ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inferencecited by 1
  23. 2026Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Modelscited by 1
  24. 2026PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understandingcited by 1
  25. 2026RUTA: Principled Visual Token Allocation via Rate-Utility Optimizationcited by 1
  26. 2026Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistencycited by 1
  27. 2025AutoComPose: Automatic Generation of Pose Transition Descriptions for Composed Pose Retrieval Using Multimodal LLMscited by 0
  28. 2025Improving MLLM Training Efficiency via Stage-Aware Sparsitycited by 0
  29. 2026LinMU: Multimodal Understanding Made Linearcited by 0
  30. 2026ConsensusDrop: Fusing Visual and Cross-Modal Saliency for Efficient Vision Language Modelscited by 0
  31. 2026Parallel In-context Learning for Large Vision Language Modelscited by 0
  32. 2026Make Your LVLM KV Cache More Lightweightcited by 0
  33. 2026Large Multimodal Model-Based Environment-Aware Mobility Managementcited by 0
  34. 2026SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answering
  2. 2025QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQA
  3. 2025VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?
  4. 2024Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information
  5. 2025AVAM: Universal Training-free Adaptive Visual Anchoring Embedded into Multimodal Large Language Model for Multi-image Question Answering
  6. 2026One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
  7. 2026Rethinking Token Reduction for Large Vision-Language Models
  8. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs