# FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression

arXiv: https://arxiv.org/abs/2411.14228 · HTML: https://mlsanity.com/paper/2411.14228/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 44 of its 51 references that are in the map.
- First posted on arXiv 21 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 44)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models](https://mlsanity.com/paper/2403.15388/) (2024)
- [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://mlsanity.com/paper/2403.18814/) (2024)
- [Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models](https://mlsanity.com/paper/2311.06607/) (2023)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models](https://mlsanity.com/paper/2311.07575/) (2023)
- [GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning](https://mlsanity.com/paper/2105.14517/) (2021)
- [ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models](https://mlsanity.com/paper/2402.11684/) (2024)

## Built on by (most cited first, 22 of 22)

- [AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understanding](https://mlsanity.com/paper/2503.12559/) (2025)
- [Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention](https://mlsanity.com/paper/2510.02912/) (2025)
- [InternVL-X: Advancing and Accelerating InternVL Series with Efficient Visual Token Compression](https://mlsanity.com/paper/2503.21307/) (2025)
- [Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inference](https://mlsanity.com/paper/2503.12926/) (2025)
- [HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit](https://mlsanity.com/paper/2602.23699/) (2026)
- [Investigating Inference-time Scaling for Chain of Multi-modal Thought: A Preliminary Study](https://mlsanity.com/paper/2502.11514/) (2025)
- [VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs](https://mlsanity.com/paper/2510.16598/) (2025)
- [Slow-Fast Architecture for Video Multi-Modal Large Language Models](https://mlsanity.com/paper/2504.01328/) (2025)
- [Connector-S: A Survey of Connectors in Multi-modal Large Language Models](https://mlsanity.com/paper/2502.11453/) (2025)
- [Beyond Intermediate States: Explaining Visual Redundancy through Language](https://mlsanity.com/paper/2503.20540/) (2025)
- [ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance](https://mlsanity.com/paper/2505.18757/) (2025)
- [LLaVA-Zip: Adaptive Visual Token Compression with Intrinsic Image Information](https://mlsanity.com/paper/2412.08771/) (2024)
- [Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning](https://mlsanity.com/paper/2505.11945/) (2025)
- [METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models](https://mlsanity.com/paper/2507.20842/) (2025)
- [EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generation](https://mlsanity.com/paper/2510.16776/) (2025)
- [Local Information Matters: Inference Acceleration For Grounded Conversation Generation Models Through Adaptive Local-Aware Token Pruning](https://mlsanity.com/paper/2503.23959/) (2025)
- [Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models](https://mlsanity.com/paper/2509.19191/) (2025)
- [An Efficient Token Compression Framework for Visual Object Tracking](https://mlsanity.com/paper/2605.08329/) (2026)
- [Prompt-Guided Prefiltering for VLM Image Compression](https://mlsanity.com/paper/2604.00314/) (2026)
- [FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding](https://mlsanity.com/paper/2608.22883/) (2026)
- [Task-Oriented Visual Feature Compression via Residual Vector Quantization for Device-Edge Multimodal Inference](https://mlsanity.com/paper/2609.37090/) (2026)
- [FoveaTree: Information-Density Trees for Query-Guided Visual Token Pruning in Video Large Language Models](https://mlsanity.com/paper/2610.16744/) (2026)

## Abstract (arXiv)

Recent advances on Multi-modal Large Language Models have demonstrated that high-resolution image input is crucial for model capabilities, especially for fine-grained tasks. However, high-resolution images lead to a quadratic increase in the number of visual tokens input into LLMs, resulting in significant computational costs. Current work develop visual token compression methods to achieve efficiency improvements, often at the expense of performance. We argue that removing visual redundancy can simultaneously improve both efficiency and performance. We build a coarse-to-fine visual token compression method, with a vision-guided sampler for compressing redundant regions with low information density, and a text-guided sampler for selecting visual tokens that are strongly correlated with the user instructions.With these two modules, the proposed FocusLLaVA achieves improvements in both efficiency and performance. We validate the effectiveness of our approach on a wide range of evaluation datasets.