mlsanityData to 14 Oct 2026

FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression

arXiv 2411.14228 ↗First posted 21 Nov 2024cs.CVcited by 22

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Recent advances on Multi-modal Large Language Models have demonstrated that high-resolution image input is crucial for model capabilities, especially for fine-grained tasks. However, high-resolution images lead to a quadratic increase in the number of visual tokens input into LLMs, resulting in significant computational costs. Current work develop visual token compression methods to achieve efficiency improvements, often at the expense of performance. We argue that removing visual redundancy can simultaneously improve both efficiency and performance. We build a coarse-to-fine visual token compression method, with a vision-guided sampler for compressing redundant regions with low information density, and a text-guided sampler for selecting visual tokens that are strongly correlated with the user instructions.With these two modules, the proposed FocusLLaVA achieves improvements in both efficiency and performance. We validate the effectiveness of our approach on a wide range of evaluation datasets.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 51 references that are in the map. First posted on arXiv 21 Nov 2024.

Builds on

44 of 51 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens7 Oct
  2. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models20 Aug
  3. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding19 Jul
  4. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models12 Jun
  5. 2024Matryoshka Query Transformer for Large Vision-Language Models29 May
  6. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference9 May
  7. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  8. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images18 Mar
  9. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
23 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  7. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  8. 2021OCR-free Document Understanding Transformercited by 407
  9. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  10. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  11. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  12. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  13. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  14. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  15. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 228
  16. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  17. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  18. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  19. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 113
  20. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  21. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72
  22. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 54
  23. 2024TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Modelscited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 6.

22 papers.

Sort:

By area:

  1. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 48
  2. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 47
  3. 2025InternVL-X: Advancing and Accelerating InternVL Series with Efficient Visual Token Compressioncited by 21
  4. 2025Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inferencecited by 18
  5. 2026HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exitcited by 18
  6. 2025Investigating Inference-time Scaling for Chain of Multi-modal Thought: A Preliminary Studycited by 11
  7. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMscited by 11
  8. 2025Slow-Fast Architecture for Video Multi-Modal Large Language Modelscited by 10
  9. 2025Connector-S: A Survey of Connectors in Multi-modal Large Language Modelscited by 7
  10. 2025Beyond Intermediate States: Explaining Visual Redundancy through Languagecited by 7
  11. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevancecited by 7
  12. 2024LLaVA-Zip: Adaptive Visual Token Compression with Intrinsic Image Informationcited by 3
10 more
  1. 2025Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuningcited by 3
  2. 2025METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Modelscited by 3
  3. 2025EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generationcited by 3
  4. 2025Local Information Matters: Inference Acceleration For Grounded Conversation Generation Models Through Adaptive Local-Aware Token Pruningcited by 2
  5. 2025Reading Images Like Texts: Sequential Image Understanding in Vision-Language Modelscited by 2
  6. 2026An Efficient Token Compression Framework for Visual Object Trackingcited by 1
  7. 2026Prompt-Guided Prefiltering for VLM Image Compressioncited by 0
  8. 2026FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decodingcited by 0
  9. 2026Task-Oriented Visual Feature Compression via Residual Vector Quantization for Device-Edge Multimodal Inferencecited by 0
  10. 2026FoveaTree: Information-Density Trees for Query-Guided Visual Token Pruning in Video Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods
  2. 2026Beyond Selection: Token Parameterization for Extreme Visual Token Compression
  3. 2025Token Sequence Compression for Efficient Multimodal Computing
  4. 2025FCoT-VL:Advancing Text-oriented Large Vision-Language Models with Efficient Visual Token Compression
  5. 2025A Survey of Token Compression for Efficient Multimodal Large Language Models
  6. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models
  7. 2025Hybrid Token Compression for Vision-Language Models
  8. 2024Visual-Word Tokenizer: Beyond Fixed Sets of Tokens in Vision Transformers