mlsanityData to 14 Oct 2026

A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends

arXiv 2507.09861 ↗First posted 14 Jul 2025cs.CV cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Visually Rich Document Understanding (VRDU) has become a pivotal area of research, driven by the need to automatically interpret documents that contain intricate visual, textual, and structural elements. Recently, Multimodal Large Language Models (MLLMs) have demonstrated significant promise in this domain, including both OCR-based and OCR-free approaches for information extraction from document images. This survey reviews recent advances in MLLM-based VRDU, highlighting emerging trends and promising research directions with a focus on two key aspects: (1) techniques for representing and integrating textual, visual, and layout features; (2) training paradigms, including pretraining, instruction tuning, and training strategies. Moreover, we address challenges such as data scarcity, handling multi-page and multilingual documents, and integrating emerging trends such as Retrieval-Augmented Generation and agentic frameworks. Our analysis offers a roadmap for advancing MLLM-based VRDU toward more scalable, reliable, and adaptable systems.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 64 of its 97 references that are in the map. First posted on arXiv 14 Jul 2025.

Builds on

64 of 97 references.

References · 64

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  7. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  8. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  9. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  10. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  11. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  12. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
38 more
  1. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  2. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  3. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  4. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  5. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  6. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 213
  7. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 195
  8. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  9. 2019Shape Robust Text Detection with Progressive Scale Expansion Networkcited by 147
  10. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  11. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 116
  12. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 114
  13. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  14. 2021BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentscited by 104
  15. 2018Chargrid: Towards Understanding 2D Documentscited by 100
  16. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 96
  17. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 95
  18. 2017Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networkcited by 92
  19. 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacycited by 81
  20. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understandingcited by 74
  21. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 72
  22. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 67
  23. 2019BERTgrid: Contextualized Embedding for 2D Document Representation and Understandingcited by 63
  24. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  25. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  26. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  27. 2023LMDX: Language Model-based Document Information Extraction and Localizationcited by 37
  28. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  29. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  30. 2025SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoningcited by 34
  31. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  32. 2024Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Modelscited by 30
  33. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  34. 2025SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinementcited by 28
  35. 2024InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructionscited by 27
  36. 2023Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMscited by 25
  37. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  38. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Benchmarking Vision-Language Models for French PDF-to-Markdown Conversioncited by 1
  2. 2026Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligencecited by 1
  3. 2026Multimodal Approaches for Visually-Rich Document Type Classification: A Comparative Analysiscited by 0
  4. 2026BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehensioncited by 0
  5. 2026XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understandingcited by 0
  6. 2026DocAtlas: Long-Document Understanding as Mutable-State Interactioncited by 0
  7. 2026Question-Guided Evidence Acquisition for Multimodal Visual Question Answeringcited by 0
  8. 2026FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Survey
  2. 2025MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding
  3. 2022VRDU: A Benchmark for Visually-rich Document Understanding
  4. 2026OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets
  5. 2024DocVLM: Make Your VLM an Efficient Reader
  6. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding
  7. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  8. 2026Visual Information Extraction from Documents via Classification-Guided Large Vision-Language Models