# Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information

arXiv: https://arxiv.org/abs/2409.01179 · HTML: https://mlsanity.com/paper/2409.01179/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models](https://mlsanity.com/paper/2403.15388/) (2024)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [DiffRate : Differentiable Compression Rate for Efficient Vision Transformers](https://mlsanity.com/paper/2305.17997/) (2023)
- [TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones](https://mlsanity.com/paper/2312.16862/) (2023)
- [Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers](https://mlsanity.com/paper/2211.11315/) (2022)
- [CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers](https://mlsanity.com/paper/2305.17455/) (2023)
- [HRVDA: High-Resolution Visual Document Assistant](https://mlsanity.com/paper/2404.06918/) (2024)
- [Question Aware Vision Transformer for Multimodal Reasoning](https://mlsanity.com/paper/2402.05472/) (2024)
- [Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers](https://mlsanity.com/paper/2403.10030/) (2024)
- [No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling](https://mlsanity.com/paper/2310.05654/) (2023)

## Built on by (most cited first, 18 of 18)

- [Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration](https://mlsanity.com/paper/2412.13180/) (2024)
- [Continual Learning for Generative AI: From LLMs to MLLMs and Beyond](https://mlsanity.com/paper/2506.13045/) (2025)
- [Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning](https://mlsanity.com/paper/2506.21873/) (2025)
- [QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQA](https://mlsanity.com/paper/2504.00654/) (2025)
- [SpecVLM: Fast Speculative Decoding in Vision-Language Models](https://mlsanity.com/paper/2509.11815/) (2025)
- [TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model](https://mlsanity.com/paper/2507.20630/) (2025)
- [Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models](https://mlsanity.com/paper/2606.12412/) (2026)
- [MeteorPred: A Meteorological Multimodal Large Model and Dataset for Severe Weather Event Prediction](https://mlsanity.com/paper/2508.06859/) (2025)
- [HVR-Met: A Hypothesis-Verification-Replanning Agentic System for Extreme Weather Diagnosis](https://mlsanity.com/paper/2603.01121/) (2026)
- [Achieving binary weight and activation for LLMs using Post-Training Quantization](https://mlsanity.com/paper/2504.05352/) (2025)
- [Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models](https://mlsanity.com/paper/2508.01236/) (2025)
- [Evaluating Variance in Visual Question Answering Benchmarks](https://mlsanity.com/paper/2508.02645/) (2025)
- [AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference with Dynamical Text Guidance](https://mlsanity.com/paper/2508.06084/) (2025)
- [Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2602.12618/) (2026)
- [Decoupled Similarity for Task-Aware Token Pruning in Large Vision-Language Models](https://mlsanity.com/paper/2604.11240/) (2026)
- [RANGER: A Monocular Zero-Shot Semantic Navigation Framework through Visual Contextual Adaptation](https://mlsanity.com/paper/2512.24212/) (2025)
- [BiNSGPS: Geometry Problem Solving via Bidirectional Neuro-Symbolic Interaction](https://mlsanity.com/paper/2606.04648/) (2026)
- [ElementCheck: Complexity-Aware Long-Form Text Factuality Evaluation via Sentence Elements](https://mlsanity.com/paper/2608.26118/) (2026)

## Abstract (arXiv)

With the advancement of large-scale language modeling techniques, large multimodal models combining visual encoders with large language models have demonstrated exceptional performance in various visual tasks. Most of the current large-scale multimodal models achieve this by mapping visual features obtained from the visual encoder into a large language model and using them as inputs alongside text for downstream tasks. Therefore, the number of visual tokens directly affects the training and inference speed of the model. There has been significant work on token pruning for visual transformers, but for large multimodal models, only relying on visual information for token pruning or compression may lead to significant loss of important information. On the other hand, the textual input in the form of a question may contain valuable information that can aid in answering the question, providing additional knowledge to the model. To address the potential oversimplification and excessive pruning that can occur with most purely visual token pruning methods, we propose a text information-guided dynamic visual token recovery mechanism that does not require training. This mechanism leverages the similarity between the question text and visual tokens to recover visually meaningful tokens with important text information while merging other less important tokens. Experimental results demonstrate that our proposed method achieves comparable performance to the original approach while compressing the visual tokens to an average of 10% of the original quantity. Our source code will be made publicly available following acceptance.