Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training
On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG
Abstract (arXiv)
Document parsing has recently advanced with multimodal large language models (MLLMs) that directly map document images to structured outputs. Traditional cascaded pipelines depend on precise layout analysis and often fail under casually captured or non-standard conditions. Although end-to-end approaches mitigate this dependency, they still exhibit repetitive, hallucinated, and structurally inconsistent predictions - primarily due to the scarcity of large-scale, high-quality full-page (document-level) end-to-end parsing data and the lack of structure-aware training strategies. To address these challenges, we propose a data-training co-design framework for robust end-to-end document parsing. A Realistic Scene Synthesis strategy constructs large-scale, structurally diverse full-page end-to-end supervision by composing layout templates with rich document elements, while a Document-Aware Training Recipe introduces progressive learning and structure-token optimization to enhance structural fidelity and decoding stability. We further build Wild-OmniDocBench, a benchmark derived from real-world captured documents for robustness evaluation. Integrated into a 1B-parameter MLLM, our method achieves superior accuracy and robustness across both scanned/digital and real-world captured scenarios. All models, data synthesis pipelines, and benchmarks will be publicly released to advance future research in document understanding.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 58 references that are in the map. First posted on arXiv 25 Mar 2026.
Builds on
45 of 58 references.
Closest ancestors · 7
Same direction, newest first
- 2025dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model2 Dec
- 2025PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model16 Oct
- 2025MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing26 Sep
- 2025POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion1 Sep
- 2025WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?16 May
- 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models25 Feb
- 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding8 Apr
Other references · 38
- 2023GPT-4 Technical Reportcited by 18,132
- 2023Visual Instruction Tuningcited by 7,792
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
- 2021OCR-free Document Understanding Transformercited by 400
- 2023Effective Long-Context Scaling of Foundation Modelscited by 301
- 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
- 2019Image-based table recognition: data, model, and evaluationcited by 233
- 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
26 more
- 2025DeepSeek-OCR: Contexts Optical Compressioncited by 174
- 2020ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Networkcited by 170
- 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 150
- 2020Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Contextcited by 146
- 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 144
- 2025PaddleOCR 3.0 Technical Reportcited by 136
- 2020Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spottingcited by 114
- 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
- 2025A Comprehensive Survey on Long Context Language Modelingcited by 113
- 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
- 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
- 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 81
- 2024DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perceptioncited by 80
- 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 78
- 2022Syntax-Aware Network for Handwritten Mathematical Expression Recognitioncited by 68
- 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
- 2023Improving Table Structure Recognition with Visual-Alignment Sequential Coordinate Modelingcited by 41
- 2024UniMERNet: A Universal Network for Real-World Mathematical Expression Recognitioncited by 41
- 2021CDistNet: Perceiving Multi-Domain Character Distance for Robust Text Recognitioncited by 36
- 2021Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformercited by 34
- 2022CoMER: Modeling Coverage for Transformer-based Handwritten Mathematical Expression Recognitioncited by 24
- 2023Context Perception Parallel Decoder for Scene Text Recognitioncited by 19
- 2024Instruction-Guided Scene Text Recognitioncited by 17
- 2023IPAD: Iterative, Parallel, and Diffusion-based Network for Scene Text Recognitioncited by 12
- 2024Arbitrary Reading Order Scene Text Spotter with Local Semantics Guidancecited by 11
- 2024LDP: Generalizing to Multilingual Visual Information Extraction by Language Decoupled Pretrainingcited by 5
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026Chronicles-OCR: A Cross-Temporal Perception Benchmark for the Evolutionary Trajectory of Chinese Characterscited by 5
- 2026ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formatscited by 2
- 2026StrucTab: A Structured Optimization Framework for Table Parsingcited by 2
- 2026MORE: A Multilingual Document Parsing Benchmark and Evaluationcited by 2
- 2026TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documentscited by 0
- 2026WeVisDoc: From Coverage to Capability for Robust End-to-End Document Parsingcited by 0
- 2026Xiaomi-OCR-0 Technical Reportcited by 0
- 2026Xiaomi-OCR-0 Technical Report28 Sep 2026
- 2026WeVisDoc: From Coverage to Capability for Robust End-to-End Document Parsing17 Sep 2026
- 2026TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents13 Aug 2026
- 2026MORE: A Multilingual Document Parsing Benchmark and Evaluation3 Jul 2026
- 2026StrucTab: A Structured Optimization Framework for Table Parsing29 Jun 2026
- 2026ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats31 May 2026
- 2026Chronicles-OCR: A Cross-Temporal Perception Benchmark for the Evolutionary Trajectory of Chinese Characters12 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild
- 2025DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation
- 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing
- 2026Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding
- 2026HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding
- 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Prompting
- 2026Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing
- 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding