# mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding

arXiv: https://arxiv.org/abs/2409.03420 · HTML: https://mlsanity.com/paper/2409.03420/

## Key facts

- Cited by 114 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 51 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Builds on (most cited first, 30 of 49)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [InfographicVQA](https://mlsanity.com/paper/2104.12756/) (2021)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://mlsanity.com/paper/2403.18814/) (2024)
- [Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models](https://mlsanity.com/paper/2311.06607/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding](https://mlsanity.com/paper/2403.12895/) (2024)

## Built on by (most cited first, 30 of 114)

- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)
- [VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents](https://mlsanity.com/paper/2410.10594/) (2024)
- [OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning](https://mlsanity.com/paper/2501.00321/) (2024)
- [OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations](https://mlsanity.com/paper/2412.07626/) (2024)
- [MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm](https://mlsanity.com/paper/2506.05218/) (2025)
- [Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting](https://mlsanity.com/paper/2505.14059/) (2025)
- [LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating](https://mlsanity.com/paper/2412.18424/) (2024)
- [SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion](https://mlsanity.com/paper/2503.11576/) (2025)
- [Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.08826/) (2025)
- [EVEv2: Improved Baselines for Encoder-Free Vision-Language Models](https://mlsanity.com/paper/2502.06788/) (2025)
- [POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion](https://mlsanity.com/paper/2509.01215/) (2025)
- [TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens](https://mlsanity.com/paper/2410.05261/) (2024)
- [Harnessing Webpage UIs for Text-Rich Visual Understanding](https://mlsanity.com/paper/2410.13824/) (2024)
- [DocVLM: Make Your VLM an Efficient Reader](https://mlsanity.com/paper/2412.08746/) (2024)
- [FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understanding](https://mlsanity.com/paper/2504.17447/) (2025)
- [Ocean-OCR: Towards General OCR Application via a Vision-Language Model](https://mlsanity.com/paper/2501.15558/) (2025)
- [Dolphin-v2: Universal Document Parsing via Scalable Anchor Prompting](https://mlsanity.com/paper/2602.05384/) (2026)
- [Vision-centric Token Compression in Large Language Model](https://mlsanity.com/paper/2502.00791/) (2025)
- [ModernVBERT: Towards Smaller Visual Document Retrievers](https://mlsanity.com/paper/2510.01149/) (2025)
- [Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understanding](https://mlsanity.com/paper/2503.14140/) (2025)
- [DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning](https://mlsanity.com/paper/2509.23883/) (2025)
- [Docopilot: Improving Multimodal Models for Document-Level Understanding](https://mlsanity.com/paper/2507.14675/) (2025)
- [Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks](https://mlsanity.com/paper/2410.01744/) (2024)
- [Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing](https://mlsanity.com/paper/2510.15349/) (2025)
- [PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Sampling](https://mlsanity.com/paper/2410.05970/) (2024)
- [A Token-level Text Image Foundation Model for Document Understanding](https://mlsanity.com/paper/2503.02304/) (2025)
- [Multimodal OCR: Parse Anything from Documents](https://mlsanity.com/paper/2603.13032/) (2026)
- [MonkeyOCR v1.5 Technical Report: Unlocking Robust Document Parsing for Complex Patterns](https://mlsanity.com/paper/2511.10390/) (2025)
- [Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skipping](https://mlsanity.com/paper/2503.21817/) (2025)
- [DocDancer: Towards Agentic Document-Grounded Information Seeking](https://mlsanity.com/paper/2601.05163/) (2026)

## Abstract (arXiv)

Multimodel Large Language Models(MLLMs) have achieved promising OCR-free Document Understanding performance by increasing the supported resolution of document images. However, this comes at the cost of generating thousands of visual tokens for a single document image, leading to excessive GPU memory and slower inference times, particularly in multi-page document comprehension. In this work, to address these challenges, we propose a High-resolution DocCompressor module to compress each high-resolution document image into 324 tokens, guided by low-resolution global visual features. With this compression module, to strengthen multi-page document comprehension ability and balance both token efficiency and question-answering performance, we develop the DocOwl2 under a three-stage training framework: Single-image Pretraining, Multi-image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new state-of-the-art across multi-page document understanding benchmarks and reduces first token latency by more than 50%, demonstrating advanced capabilities in multi-page questioning answering, explanation with evidence pages, and cross-page structure understanding. Additionally, compared to single-image MLLMs trained on similar data, our DocOwl2 achieves comparable single-page understanding performance with less than 20% of the visual tokens. Our codes, models, and data are publicly available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2.