# Document Understanding Dataset and Evaluation (DUDE)

arXiv: https://arxiv.org/abs/2305.08455 · HTML: https://mlsanity.com/paper/2305.08455/

## Key facts

- Cited by 135 papers in the map (data through 14 Oct 2026).
- Builds on 78 of its 117 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 30 of 78)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [InfographicVQA](https://mlsanity.com/paper/2104.12756/) (2021)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning](https://mlsanity.com/paper/2105.14517/) (2021)
- [ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning](https://mlsanity.com/paper/2002.04326/) (2020)

## Built on by (most cited first, 30 of 135)

- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning](https://mlsanity.com/paper/2403.16999/) (2024)
- [MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations](https://mlsanity.com/paper/2407.01523/) (2024)
- [OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning](https://mlsanity.com/paper/2501.00321/) (2024)
- [NVLM: Open Frontier-Class Multimodal LLMs](https://mlsanity.com/paper/2409.11402/) (2024)
- [DocLLM: A layout-aware generative language model for multimodal document understanding](https://mlsanity.com/paper/2401.00908/) (2023)
- [mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding](https://mlsanity.com/paper/2409.03420/) (2024)
- [M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding](https://mlsanity.com/paper/2411.04952/) (2024)
- [LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating](https://mlsanity.com/paper/2412.18424/) (2024)
- [Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models](https://mlsanity.com/paper/2504.15271/) (2025)
- [VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents](https://mlsanity.com/paper/2504.09795/) (2025)
- [Seed1.8 Model Card: Towards Generalized Real-World Agency](https://mlsanity.com/paper/2603.20633/) (2026)
- [Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2505.20272/) (2025)
- [MMDocIR: Benchmarking Multimodal Retrieval for Long Documents](https://mlsanity.com/paper/2501.08828/) (2025)
- [VisRL: Intention-Driven Visual Perception via Reinforced Reasoning](https://mlsanity.com/paper/2503.07523/) (2025)
- [Overcoming Common Flaws in the Evaluation of Selective Classification Systems](https://mlsanity.com/paper/2407.01032/) (2024)
- [DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models](https://mlsanity.com/paper/2406.11633/) (2024)
- [Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering](https://mlsanity.com/paper/2505.16470/) (2025)
- [PDFTriage: Question Answering over Long, Structured Documents](https://mlsanity.com/paper/2309.08872/) (2023)
- [DocVLM: Make Your VLM an Efficient Reader](https://mlsanity.com/paper/2412.08746/) (2024)
- [SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinement](https://mlsanity.com/paper/2506.14035/) (2025)
- [A Survey on Multimodal Benchmarks: In the Era of Large AI Models](https://mlsanity.com/paper/2409.18142/) (2024)
- [OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.02592/) (2024)
- [Introducing Visual Perception Token into Multimodal Large Language Model](https://mlsanity.com/paper/2502.17425/) (2025)
- [ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios](https://mlsanity.com/paper/2601.08620/) (2026)
- [SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding](https://mlsanity.com/paper/2411.01106/) (2024)
- [GRAM: Global Reasoning for Multi-Page VQA](https://mlsanity.com/paper/2401.03411/) (2024)
- [Deep Learning based Visually Rich Document Content Understanding: A Survey](https://mlsanity.com/paper/2408.01287/) (2024)
- [MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding](https://mlsanity.com/paper/2410.21311/) (2024)

## Abstract (arXiv)

We call on the Document AI (DocAI) community to reevaluate current methodologies and embrace the challenge of creating more practically-oriented benchmarks. Document Understanding Dataset and Evaluation (DUDE) seeks to remediate the halted research progress in understanding visually-rich documents (VRDs). We present a new dataset with novelties related to types of questions, answers, and document layouts based on multi-industry, multi-domain, and multi-page VRDs of various origins, and dates. Moreover, we are pushing the boundaries of current methods by creating multi-task and multi-domain evaluation setups that more accurately simulate real-world situations where powerful generalization and adaptation under low-resource settings are desired. DUDE aims to set a new standard as a more practical, long-standing benchmark for the community, and we hope that it will lead to future extensions and contributions that address real-world challenges. Finally, our work illustrates the importance of finding more efficient ways to model language, images, and layout in DocAI.