# READoc: A Unified Benchmark for Realistic Document Structured Extraction

arXiv: https://arxiv.org/abs/2409.05137 · HTML: https://mlsanity.com/paper/2409.05137/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 49 references that are in the map.
- First posted on arXiv 8 Sep 2024.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models](https://mlsanity.com/paper/2311.06607/) (2023)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding](https://mlsanity.com/paper/2403.12895/) (2024)
- [CORD-19: The COVID-19 Open Research Dataset](https://mlsanity.com/paper/2004.10706/) (2020)
- [General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model](https://mlsanity.com/paper/2409.01704/) (2024)
- [TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document](https://mlsanity.com/paper/2403.04473/) (2024)
- [DocBank: A Benchmark Dataset for Document Layout Analysis](https://mlsanity.com/paper/2006.01038/) (2020)
- [PubTables-1M: Towards comprehensive table extraction from unstructured documents](https://mlsanity.com/paper/2110.00061/) (2021)
- [SciREX: A Challenge Dataset for Document-Level Information Extraction](https://mlsanity.com/paper/2005.00512/) (2020)
- [DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding](https://mlsanity.com/paper/2311.11810/) (2023)
- [KOSMOS-2.5: A Multimodal Literate Model](https://mlsanity.com/paper/2309.11419/) (2023)
- [LayoutReader: Pre-training of Text and Layout for Reading Order Detection](https://mlsanity.com/paper/2108.11591/) (2021)
- [Image-to-Markup Generation with Coarse-to-Fine Attention](https://mlsanity.com/paper/1609.04938/) (2016)
- [TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy](https://mlsanity.com/paper/2406.01326/) (2024)
- [Focus Anywhere for Fine-grained Multi-page Document Understanding](https://mlsanity.com/paper/2405.14295/) (2024)
- [Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages](https://mlsanity.com/paper/2308.12038/) (2023)
- [Small Language Model Meets with Reinforced Vision Vocabulary](https://mlsanity.com/paper/2401.12503/) (2024)
- [VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups](https://mlsanity.com/paper/2106.00676/) (2021)
- [HRDoc: Dataset and Baseline Method Toward Hierarchical Reconstruction of Document Structures](https://mlsanity.com/paper/2303.13839/) (2023)

## Built on by (most cited first, 15 of 15)

- [OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations](https://mlsanity.com/paper/2412.07626/) (2024)
- [OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.02592/) (2024)
- [XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaboration](https://mlsanity.com/paper/2505.11336/) (2025)
- [CC-OCR V2: Fine-Grained Attribution of LMM Failures in Real-World Visual Document Understanding](https://mlsanity.com/paper/2605.03903/) (2026)
- [Beyond String Matching: Semantic Evaluation of PDF Table Extraction](https://mlsanity.com/paper/2603.18652/) (2026)
- [From PDF to RAG-Ready: Evaluating Document Conversion Frameworks for Domain-Specific Question Answering](https://mlsanity.com/paper/2604.04948/) (2026)
- [TexOCR: Advancing Document OCR Models for Compilable Page-to-LaTeX Reconstruction](https://mlsanity.com/paper/2604.22880/) (2026)
- [DocAtlas: Multilingual Document Understanding Across 80+ Languages](https://mlsanity.com/paper/2605.12623/) (2026)
- [MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing](https://mlsanity.com/paper/2605.22100/) (2026)
- [Chronos: The AI Co-Historian](https://mlsanity.com/paper/2604.03553/) (2026)
- [Benchmarking Complex Multimodal Document Processing Pipelines: A Unified Evaluation Framework for Enterprise AI](https://mlsanity.com/paper/2604.26382/) (2026)
- [A Dual-Track Framework for Template-Constrained LaTeX Conversion](https://mlsanity.com/paper/2606.23107/) (2026)
- [LongDocBench: Benchmarking TOC Hierarchy and Contextual Relationship Recovery in Long Documents](https://mlsanity.com/paper/2608.15064/) (2026)
- [Code as Representation: A Compilable Parsing Paradigm for Academic Documents](https://mlsanity.com/paper/2608.17550/) (2026)
- [How Far Can Synthetic Data Take Thai OCR?](https://mlsanity.com/paper/2609.03595/) (2026)

## Abstract (arXiv)

Document Structured Extraction (DSE) aims to extract structured content from raw documents. Despite the emergence of numerous DSE systems, their unified evaluation remains inadequate, significantly hindering the field's advancement. This problem is largely attributed to existing benchmark paradigms, which exhibit fragmented and localized characteristics. To address these limitations and offer a thorough evaluation of DSE systems, we introduce a novel benchmark named READoc, which defines DSE as a realistic task of converting unstructured PDFs into semantically rich Markdown. The READoc dataset is derived from 3,576 diverse and real-world documents from arXiv, GitHub, and Zenodo. In addition, we develop a DSE Evaluation S3uite comprising Standardization, Segmentation and Scoring modules, to conduct a unified evaluation of state-of-the-art DSE approaches. By evaluating a range of pipeline tools, expert visual models, and general VLMs, we identify the gap between current work and the unified, realistic DSE objective for the first time. We aspire that READoc will catalyze future research in DSE, fostering more comprehensive and practical solutions.