mlsanityData to 7 Oct 2026

mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding

arXiv 2403.12895 ↗First posted 19 Mar 2024cs.CVcited by 220

Not on the map. Why?

Cited by 113 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Structure information is critical for understanding the semantics of text-rich images, such as documents, tables, and charts. Existing Multimodal Large Language Models (MLLMs) for Visual Document Understanding are equipped with text recognition ability but lack general structure understanding abilities for text-rich document images. In this work, we emphasize the importance of structure information in Visual Document Understanding and propose the Unified Structure Learning to boost the performance of MLLMs. Our Unified Structure Learning comprises structure-aware parsing tasks and multi-grained text localization tasks across 5 domains: document, webpage, table, chart, and natural image. To better encode structure information, we design a simple and effective vision-to-text module H-Reducer, which can not only maintain the layout information but also reduce the length of visual features by merging horizontal adjacent patches through convolution, enabling the LLM to understand high-resolution images more efficiently. Furthermore, by constructing structure-aware text sequences and multi-grained pairs of texts and bounding boxes for publicly available text-rich images, we build a comprehensive training set DocStruct4M to support structure learning. Finally, we construct a small but high-quality reasoning tuning dataset DocReason25K to trigger the detailed explanation ability in the document domain. Our model DocOwl 1.5 achieves state-of-the-art performance on 10 visual document understanding benchmarks, improving the SOTA performance of MLLMs with a 7B LLM by more than 10 points in 5/10 benchmarks. Our codes, models, and datasets are publicly available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl1.5.

Key facts. Cited by 220 papers in the map (data through 7 Oct 2026). Builds on 58 of its 70 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

58 of 70 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023A Survey of Large Language Modelscited by 2,537
38 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  12. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  13. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  14. 2021InfographicVQAcited by 430
  15. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  16. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  17. 2021OCR-free Document Understanding Transformercited by 400
  18. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  19. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  20. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  21. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  22. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  23. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  24. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  25. 2019Image-based table recognition: data, model, and evaluationcited by 233
  26. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  27. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  28. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  29. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  30. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  31. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  32. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  33. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  34. 2023TableLlama: Towards Open Large Generalist Models for Tablescited by 136
  35. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 111
  36. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  37. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  38. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 77; 2025: 95; 2026: 48.

220 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  3. 2024What matters when building vision-language models?cited by 321
  4. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  5. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  6. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  7. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  8. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  9. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  10. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  11. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  12. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
38 more
  1. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  2. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  3. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  4. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  5. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 144
  6. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  7. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 116
  8. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  9. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  10. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  11. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  12. 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacycited by 80
  13. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  14. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  15. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  16. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 70
  17. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  18. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 66
  19. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  20. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 61
  21. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  22. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
  23. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  24. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  25. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 51
  26. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 44
  27. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractioncited by 44
  28. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  29. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  30. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  31. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  32. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  33. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 34
  34. 2025POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversioncited by 34
  35. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  36. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  37. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMscited by 30
  38. 2024Harnessing Webpage UIs for Text-Rich Visual Understandingcited by 28

50 most cited of 220 All 220 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding
  2. 2026Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training
  3. 2026TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
  4. 2026PARL: Position-Aware Relation Learning Network for Document Layout Analysis
  5. 2024HRVDA: High-Resolution Visual Document Assistant
  6. 2026Qianfan-OCR: A Unified End-to-End Model for Document Intelligence
  7. 2026Visual Information Extraction from Documents via Classification-Guided Large Vision-Language Models
  8. 2025Index-Preserving Lightweight Token Pruning for Efficient Document Understanding in Vision-Language Models