mlsanityData to 7 Oct 2026

mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding

arXiv 2409.03420 ↗First posted 5 Sep 2024cs.CVcited by 113

Not on the map. Why?

Cited by 56 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Multimodel Large Language Models(MLLMs) have achieved promising OCR-free Document Understanding performance by increasing the supported resolution of document images. However, this comes at the cost of generating thousands of visual tokens for a single document image, leading to excessive GPU memory and slower inference times, particularly in multi-page document comprehension. In this work, to address these challenges, we propose a High-resolution DocCompressor module to compress each high-resolution document image into 324 tokens, guided by low-resolution global visual features. With this compression module, to strengthen multi-page document comprehension ability and balance both token efficiency and question-answering performance, we develop the DocOwl2 under a three-stage training framework: Single-image Pretraining, Multi-image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new state-of-the-art across multi-page document understanding benchmarks and reduces first token latency by more than 50%, demonstrating advanced capabilities in multi-page questioning answering, explanation with evidence pages, and cross-page structure understanding. Additionally, compared to single-image MLLMs trained on similar data, our DocOwl2 achieves comparable single-page understanding performance with less than 20% of the visual tokens. Our codes, models, and data are publicly available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Builds on 49 of its 51 references that are in the map. First posted on arXiv 5 Sep 2024.

Builds on

49 of 51 references.

References · 49

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  12. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
37 more
  1. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  2. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  3. 2024Long Context Transfer from Language to Visioncited by 502
  4. 2021InfographicVQAcited by 430
  5. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  6. 2021OCR-free Document Understanding Transformercited by 400
  7. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  8. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  9. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  10. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  11. 2023Adapting Language Models to Compress Contextscited by 350
  12. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  13. 2024What matters when building vision-language models?cited by 321
  14. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  15. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  16. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  17. 2019Image-based table recognition: data, model, and evaluationcited by 233
  18. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  19. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  20. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  21. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  22. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  23. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  24. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  25. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  26. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  27. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  28. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 123
  29. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  30. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  31. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  32. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  33. 2021Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layoutscited by 82
  34. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 34
  35. 2022Watching the News: Towards VideoQA Models that can Readcited by 20
  36. 2023MPMQA: Multimodal Question Answering on Product Manualscited by 10
  37. 2021Question-controlled Text-aware Image Captioningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 59; 2026: 40.

113 papers.

Sort:

By area:

  1. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  2. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  3. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  4. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 144
  5. 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 81
  6. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 78
  7. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 70
  8. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 51
  9. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  10. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 38
  11. 2025POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversioncited by 34
  12. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
38 more
  1. 2024Harnessing Webpage UIs for Text-Rich Visual Understandingcited by 28
  2. 2024DocVLM: Make Your VLM an Efficient Readercited by 27
  3. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 26
  4. 2025Ocean-OCR: Towards General OCR Application via a Vision-Language Modelcited by 24
  5. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  6. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  7. 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Promptingcited by 20
  8. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  9. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  10. 2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 19
  11. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  12. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 17
  13. 2024PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Samplingcited by 14
  14. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  15. 2025MonkeyOCR v1.5 Technical Report: Unlocking Robust Document Parsing for Complex Patternscited by 13
  16. 2026Multimodal OCR: Parse Anything from Documentscited by 12
  17. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
  18. 2025Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skippingcited by 11
  19. 2025DocR1: Evidence Page-Guided GRPO for Multi-Page Document Understandingcited by 11
  20. 2026DocDancer: Towards Agentic Document-Grounded Information Seekingcited by 11
  21. 2024DOGR: Towards Versatile Visual Document Grounding and Referringcited by 10
  22. 2025PosterSum: A Multimodal Benchmark for Scientific Poster Summarizationcited by 10
  23. 2025HiPrune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Modelscited by 10
  24. 2025A Simple yet Effective Layout Token in Large Language Models for Document Understandingcited by 9
  25. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  26. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  27. 2025One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Imagecited by 8
  28. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  29. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMscited by 7
  30. 2026Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Trainingcited by 7
  31. 2025Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?cited by 6
  32. 2025Chart-HQA: A Benchmark for Hypothetical Question Answering in Chartscited by 5
  33. 2025MMCR: Benchmarking Cross-Source Reasoning in Scientific Paperscited by 5
  34. 2025Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiencycited by 5
  35. 2025TABLET: A Large-Scale Dataset for Robust Visual Table Understandingcited by 5
  36. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  37. 2025URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understandingcited by 5
  38. 2025Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmarkcited by 5

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024GRAM: Global Reasoning for Multi-Page VQA
  2. 2026DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruning
  3. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding
  4. 2026Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA
  5. 2026A Table Is Worth 64 Tokens: Pixel-level Compression for Multi-Table Document Question Answering
  6. 2025Interpret, prune and distill Donut : towards lightweight VLMs for VQA on document
  7. 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents
  8. 2024HRVDA: High-Resolution Visual Document Assistant