mlsanityData to 7 Oct 2026

Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction

arXiv 2410.21169 ↗First posted 28 Oct 2024cs.MM cs.AI cs.CLcited by 44

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Document parsing (DP) transforms unstructured or semi-structured documents into structured, machine-readable representations, enabling downstream applications such as knowledge base construction and retrieval-augmented generation (RAG). This survey provides a comprehensive and timely review of document parsing research. We propose a systematic taxonomy that organizes existing approaches into modular pipeline-based systems and unified models driven by Vision-Language Models (VLMs). We provide a detailed review of key components in pipeline systems, including layout analysis and the recognition of heterogeneous content such as text, tables, mathematical expressions, and visual elements, and then systematically track the evolution of specialized VLMs for document parsing. Additionally, we summarize widely adopted evaluation metrics and high-quality benchmarks that establish current standards for parsing quality. Finally, we discuss key open challenges, including robustness to complex layouts, reliability of VLM-based parsing, and inference efficiency, and outline directions for building more accurate and scalable document intelligence systems.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 96 of its 252 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

96 of 252 references.

References · 96

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  7. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  8. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  9. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  10. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  11. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  12. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
38 more
  1. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  2. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 398
  3. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  4. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  5. 2019PubLayNet: largest dataset ever for document layout analysiscited by 349
  6. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 278
  7. 2019Character Region Awareness for Text Detectioncited by 273
  8. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  9. 2017EAST: An Efficient and Accurate Scene Text Detectorcited by 248
  10. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  11. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  12. 2019Image-based table recognition: data, model, and evaluationcited by 233
  13. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  14. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
  15. 2020DocBank: A Benchmark Dataset for Document Layout Analysiscited by 152
  16. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  17. 2020Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Contextcited by 146
  18. 2022DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysiscited by 146
  19. 2019ICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboardcited by 127
  20. 2021PubTables-1M: Towards comprehensive table extraction from unstructured documentscited by 122
  21. 2016Robust Scene Text Recognition with Automatic Rectificationcited by 121
  22. 2016TextBoxes: A Fast Text Detector with a Single Deep Neural Networkcited by 119
  23. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  24. 2019TableBank: A Benchmark Dataset for Table Detection and Recognitioncited by 102
  25. 2018Chargrid: Towards Understanding 2D Documentscited by 100
  26. 2020CascadeTabNet: An approach for end to end table detection and structure recognition from image-based documentscited by 98
  27. 2021From Two to One: A New Scene Text Recognizer with Visual Language Modeling Networkcited by 97
  28. 2022TableFormer: Table Structure Understanding with Transformerscited by 93
  29. 2017Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networkcited by 92
  30. 2019Complicated Table Structure Recognitioncited by 91
  31. 2024DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perceptioncited by 80
  32. 2016Image-to-Markup Generation with Coarse-to-Fine Attentioncited by 79
  33. 2022SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognitioncited by 70
  34. 2020SCATTER: Selective Context Attentional Scene Text Recognizercited by 69
  35. 2022Syntax-Aware Network for Handwritten Mathematical Expression Recognitioncited by 68
  36. 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Areacited by 68
  37. 2019BERTgrid: Contextualized Embedding for 2D Document Representation and Understandingcited by 63
  38. 2018PixelLink: Detecting Scene Text via Instance Segmentationcited by 60

50 most cited of 96 All 96 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 15; 2026: 27.

44 papers.

Sort:

By area:

  1. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 144
  2. 2026MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scalecited by 26
  3. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 24
  4. 2025HunyuanOCR Technical Reportcited by 18
  5. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 17
  6. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  7. 2026HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Bettercited by 14
  8. 2025UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameterscited by 12
  9. 2026Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decodingcited by 9
  10. 2025Easy Dataset: A Unified and Extensible Framework for Synthesizing LLM Fine-Tuning Data from Unstructured Documentscited by 6
  11. 2025DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLMcited by 6
  12. 2026Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspectivecited by 6
32 more
  1. 2025Hybrid OCR-LLM Framework for Enterprise-Scale Document Information Extraction Under Copy-heavy Taskcited by 5
  2. 2026HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decodingcited by 5
  3. 2026AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generationcited by 5
  4. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  5. 2025ThaiOCRBench: A Task-Diverse Benchmark for Vision-Language Understanding in Thaicited by 4
  6. 2025TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognitioncited by 4
  7. 2025Benchmarking Document Parsers on Mathematical Formula Extraction from PDFscited by 4
  8. 2026Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representationscited by 3
  9. 2026Efficient Document Parsing via Parallel Token Predictioncited by 2
  10. 2026P-MTP: Efficient Document Parsing via Multi-Token Prediction with Progressive Depth Scalingcited by 2
  11. 2026StrucTab: A Structured Optimization Framework for Table Parsingcited by 2
  12. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 1
  13. 2025Vision-Enabled LLMs in Historical Lexicography: Digitising and Enriching Estonian-German Dictionaries from the 17th and 18th Centuriescited by 1
  14. 2026PARL: Position-Aware Relation Learning Network for Document Layout Analysiscited by 1
  15. 2025From Chaos to Clarity: Schema-Constrained AI for Auditable Biomedical Evidence Extraction from Full-Text PDFscited by 1
  16. 2026LingDT-VL-OCR: Structure-Aware Document-Level Parsing with Fine-Grained Visual Referencecited by 1
  17. 2026Beyond String Matching: Semantic Evaluation of PDF Table Extractioncited by 1
  18. 2026MADP: A Multi-Agent Pipeline for Sustainable Document Processing with Human-in-the-Loopcited by 1
  19. 2026FinixDoc: Rethinking Financial Document Parsing Beyond Saturated Benchmarkscited by 1
  20. 2025A document is worth a structured record: Principled inductive bias design for document recognitioncited by 0
  21. 2026Chronos: The AI Co-Historiancited by 0
  22. 2026RADIANT-LLM: an Agentic Retrieval Augmented Generation Framework for Reliable Decision Support in Safety-Critical Nuclear Engineeringcited by 0
  23. 2026Prefix-Adaptive Block Diffusion for Efficient Document Recognitioncited by 0
  24. 2026Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrievalcited by 0
  25. 2026PereStruct: Multimodal Semantic Assembly for Robust Historical Document Parsingcited by 0
  26. 2026Benchmarking Large Language Models for Safety Data Extractioncited by 0
  27. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  28. 2026DocAnnot -- Accelerating the Creation of Key Information Extraction Datasets with GenAI-Powered Auto-annotationcited by 0
  29. 2026Automated Tree Knowledge Graph Construction using Ontology Expansion and Retrieval from Vietnamese History Textbookscited by 0
  30. 2026Robust Hierarchical Structures for Agentic Document Analysiscited by 0
  31. 2026PolyOCR-Venus: Unified OCR Foundation Models for Text-Centric Visual Intelligencecited by 0
  32. 2026Can large language models unlock discrete data in ophthalmic diagnostic reports?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing
  2. 2026NovaLAD: A Fast, CPU-Optimized Document Extraction Pipeline for Generative AI and Data Intelligence
  3. 2026TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
  4. 2026ExStrucTiny: A Benchmark for Schema-Variable Structured Information Extraction from Document Images
  5. 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Prompting
  6. 2025dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model
  7. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models
  8. 2026MinerU-Popo: Universal Post-Processing Model for Structured Document Parsing