# Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding

arXiv: https://arxiv.org/abs/2407.14439 · HTML: https://mlsanity.com/paper/2407.14439/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 46 of its 50 references that are in the map.
- First posted on arXiv 19 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [InfographicVQA](https://mlsanity.com/paper/2104.12756/) (2021)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models](https://mlsanity.com/paper/2403.15388/) (2024)
- [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models](https://mlsanity.com/paper/2311.06607/) (2023)
- [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding](https://mlsanity.com/paper/2403.12895/) (2024)
- [LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images](https://mlsanity.com/paper/2403.11703/) (2024)
- [TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document](https://mlsanity.com/paper/2403.04473/) (2024)

## Built on by (most cited first, 22 of 22)

- [Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention](https://mlsanity.com/paper/2510.02912/) (2025)
- [HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models](https://mlsanity.com/paper/2408.10945/) (2024)
- [LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs](https://mlsanity.com/paper/2506.21862/) (2025)
- [A Survey of Token Compression for Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2507.20198/) (2025)
- [FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression](https://mlsanity.com/paper/2411.14228/) (2024)
- [FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registers](https://mlsanity.com/paper/2501.16297/) (2025)
- [Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understanding](https://mlsanity.com/paper/2503.14140/) (2025)
- [PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Records](https://mlsanity.com/paper/2601.09636/) (2026)
- [A Token-level Text Image Foundation Model for Document Understanding](https://mlsanity.com/paper/2503.02304/) (2025)
- [PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning](https://mlsanity.com/paper/2507.08064/) (2025)
- [freePruner: A Training-free Approach for Large Multimodal Model Acceleration](https://mlsanity.com/paper/2411.15446/) (2024)
- [Beyond Intermediate States: Explaining Visual Redundancy through Language](https://mlsanity.com/paper/2503.20540/) (2025)
- [LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression](https://mlsanity.com/paper/2509.00419/) (2025)
- [QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQA](https://mlsanity.com/paper/2504.00654/) (2025)
- [ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Task](https://mlsanity.com/paper/2503.04444/) (2025)
- [AVAM: Universal Training-free Adaptive Visual Anchoring Embedded into Multimodal Large Language Model for Multi-image Question Answering](https://mlsanity.com/paper/2508.17860/) (2025)
- [LFTR: Learning-Free Token Reduction for Multimodal Large Language Models](https://mlsanity.com/paper/2501.17391/) (2025)
- [Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning](https://mlsanity.com/paper/2505.18603/) (2025)
- [An Efficient Token Compression Framework for Visual Object Tracking](https://mlsanity.com/paper/2605.08329/) (2026)
- [DocCogito: Aligning Layout Cognition and Step-Level Grounded Reasoning for Document Understanding](https://mlsanity.com/paper/2603.07494/) (2026)
- [VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions](https://mlsanity.com/paper/2603.23495/) (2026)
- [AdaThinking-E: One-Token Entropy Regulation for Adaptive Thinking](https://mlsanity.com/paper/2608.26141/) (2026)

## Abstract (arXiv)

Cropping high-resolution document images into multiple sub-images is the most widely used approach for current Multimodal Large Language Models (MLLMs) to do document understanding. Most of current document understanding methods preserve all tokens within sub-images and treat them equally. This neglects their different informativeness and leads to a significant increase in the number of image tokens. To perform a more adaptive and efficient document understanding, we propose Token-level Correlation-guided Compression, a parameter-free and plug-and-play methodology to optimize token processing. Firstly, we propose an innovative approach for assessing the pattern repetitiveness based on the correlation between each patch tokens. This method identifies redundant tokens, allowing for the determination of the sub-image's information density. Secondly, we present a token-level sampling method that efficiently captures the most informative tokens by delving into the correlation between the [CLS] token and patch tokens. By integrating these strategies, we develop a plug-and-play adaptive compressor module that can be seamlessly incorporated into MLLMs utilizing cropping techniques. This module not only enhances the processing speed during training and inference but also maintains comparable performance. We conduct experiments with the SOTA document understanding model mPLUG-DocOwl1.5 and the effectiveness is demonstrated through extensive comparisons with other compression methods.