mlsanityData to 7 Oct 2026

mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding

arXiv 2307.02499 ↗First posted 4 Jul 2023cs.CL cs.AIcited by 167

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Document understanding refers to automatically extract, analyze and comprehend information from various types of digital documents, such as a web page. Existing Multi-model Large Language Models (MLLMs), including mPLUG-Owl, have demonstrated promising zero-shot capabilities in shallow OCR-free text recognition, indicating their potential for OCR-free document understanding. Nevertheless, without in-domain training, these models tend to ignore fine-grained OCR features, such as sophisticated tables or large blocks of text, which are essential for OCR-free document understanding. In this paper, we propose mPLUG-DocOwl based on mPLUG-Owl for OCR-free document understanding. Specifically, we first construct a instruction tuning dataset featuring a wide range of visual-text understanding tasks. Then, we strengthen the OCR-free document understanding ability by jointly train the model on language-only, general vision-and-language, and document instruction tuning dataset with our unified instruction tuning strategy. We also build an OCR-free document instruction understanding evaluation set LLMDoc to better compare models' capabilities on instruct compliance and document understanding. Experimental results show that our model outperforms existing multi-modal models, demonstrating its strong ability of document understanding. Besides, without specific fine-tuning, mPLUG-DocOwl generalizes well on various downstream tasks. Our code, models, training data and evaluation set are available at https://github.com/X-PLUG/mPLUG-DocOwl.

Key facts. Cited by 167 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 37 references that are in the map. First posted on arXiv 4 Jul 2023.

Builds on

29 of 37 references.

References · 29

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
17 more
  1. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  2. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  3. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  4. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  5. 2021InfographicVQAcited by 430
  6. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  7. 2021OCR-free Document Understanding Transformercited by 400
  8. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  9. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  10. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  11. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  12. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  13. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  14. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  15. 2021Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layoutscited by 82
  16. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  17. 2021Question-controlled Text-aware Image Captioningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 72; 2025: 46; 2026: 27.

167 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  5. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  6. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  7. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  8. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  9. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  10. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  11. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
  12. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
38 more
  1. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  2. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  3. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  4. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  5. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  6. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 119
  7. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  8. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  9. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  10. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  11. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  12. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  13. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94
  14. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  15. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  16. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  17. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  18. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  19. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 78
  20. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understandingcited by 74
  21. 2023AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZcited by 74
  22. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 70
  23. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  24. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  25. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  26. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  27. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  28. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  29. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  30. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  31. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  32. 2025Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detectorcited by 38
  33. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  34. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  35. 2024From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Taskscited by 37
  36. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  37. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  38. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35

50 most cited of 167 All 167 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding
  2. 2026DocOCR-Eval: A Correction-Based Framework for OCR Tool Selection Without Ground Truth
  3. 2025PP-DocBee: Improving Multimodal Document Understanding Through a Bag of Tricks
  4. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
  5. 2026DocAtlas: Multilingual Document Understanding Across 80+ Languages
  6. 2026Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
  7. 2026Visual Information Extraction from Documents via Classification-Guided Large Vision-Language Models
  8. 2023Efficient End-to-End Visual Document Understanding with Rationale Distillation