# VRDU: A Benchmark for Visually-rich Document Understanding

arXiv: https://arxiv.org/abs/2211.15421 · HTML: https://mlsanity.com/paper/2211.15421/

## Key facts

- Cited by 35 papers in the map (data through 7 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction](https://mlsanity.com/paper/2103.10213/) (2021)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [DocBank: A Benchmark Dataset for Document Layout Analysis](https://mlsanity.com/paper/2006.01038/) (2020)
- [Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer](https://mlsanity.com/paper/2102.09550/) (2021)
- [LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding](https://mlsanity.com/paper/2202.13669/) (2022)
- [Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layouts](https://mlsanity.com/paper/2105.05796/) (2021)
- [LaTr: Layout-Aware Transformer for Scene-Text VQA](https://mlsanity.com/paper/2112.12494/) (2021)
- [FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction](https://mlsanity.com/paper/2203.08411/) (2022)
- [TRIE: End-to-End Text Reading and Information Extraction for Document Understanding](https://mlsanity.com/paper/2005.13118/) (2020)
- [LAMBERT: Layout-Aware (Language) Modeling for information extraction](https://mlsanity.com/paper/2002.08087/) (2020)
- [ROPE: Reading Order Equivariant Positional Encoding for Graph-based Document Information Extraction](https://mlsanity.com/paper/2106.10786/) (2021)

## Built on by (most cited first, 30 of 35)

- [Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills](https://mlsanity.com/paper/2603.25158/) (2026)
- [DocLLM: A layout-aware generative language model for multimodal document understanding](https://mlsanity.com/paper/2401.00908/) (2023)
- [MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing](https://mlsanity.com/paper/2509.22186/) (2025)
- [OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation](https://mlsanity.com/paper/2407.19056/) (2024)
- [LMDX: Language Model-based Document Information Extraction and Localization](https://mlsanity.com/paper/2309.10952/) (2023)
- [DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models](https://mlsanity.com/paper/2406.11633/) (2024)
- [Deep Learning based Visually Rich Document Content Understanding: A Survey](https://mlsanity.com/paper/2408.01287/) (2024)
- [Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understanding](https://mlsanity.com/paper/2409.19672/) (2024)
- [ANLS* -- A Universal Document Processing Metric for Generative Large Language Models](https://mlsanity.com/paper/2402.03848/) (2024)
- [BuDDIE: A Business Document Dataset for Multi-task Information Extraction](https://mlsanity.com/paper/2404.04003/) (2024)
- [BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations](https://mlsanity.com/paper/2501.03403/) (2025)
- [ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction](https://mlsanity.com/paper/2602.12247/) (2026)
- [Deep Learning based Key Information Extraction from Business Documents: Systematic Literature Review](https://mlsanity.com/paper/2408.06345/) (2024)
- [Problem Solved? Information Extraction Design Space for Layout-Rich Documents using LLMs](https://mlsanity.com/paper/2502.18179/) (2025)
- [KVP10k : A Comprehensive Dataset for Key-Value Pair Extraction in Business Documents](https://mlsanity.com/paper/2405.00505/) (2024)
- [Document Intelligence in the Era of Large Language Models: A Survey](https://mlsanity.com/paper/2510.13366/) (2025)
- [RealKIE: Five Novel Datasets for Enterprise Key Information Extraction](https://mlsanity.com/paper/2403.20101/) (2024)
- [IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluation](https://mlsanity.com/paper/2505.13498/) (2025)
- [VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents](https://mlsanity.com/paper/2603.15118/) (2026)
- [ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents](https://mlsanity.com/paper/2510.25668/) (2025)
- [Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning](https://mlsanity.com/paper/2505.18603/) (2025)
- [ExStrucTiny: A Benchmark for Schema-Variable Structured Information Extraction from Document Images](https://mlsanity.com/paper/2602.12203/) (2026)
- [CC-OCR V2: Fine-Grained Attribution of LMM Failures in Real-World Visual Document Understanding](https://mlsanity.com/paper/2605.03903/) (2026)
- [Pre-Inference Routing for Cost-Efficient Document Field Extraction](https://mlsanity.com/paper/2608.06607/) (2026)
- [Towards Multi-Task Multi-Modal Models: A Video Generative Perspective](https://mlsanity.com/paper/2405.16728/) (2024)
- [SynDoc: A Hybrid Discriminative-Generative Framework for Enhancing Synthetic Domain-Adaptive Document Key Information Extraction](https://mlsanity.com/paper/2509.23273/) (2025)
- [Docs2Synth: A Synthetic Data Trained Retriever Framework for Scanned Visually Rich Documents Understanding](https://mlsanity.com/paper/2601.12260/) (2026)
- [ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction](https://mlsanity.com/paper/2607.29677/) (2026)
- [D2-ScaleAgent: Dual-Dimensional Scaling for Long Document Understanding](https://mlsanity.com/paper/2608.16417/) (2026)
- [LAVA: Logic-Aware Validation and Augmentation Framework for Large-Scale Financial Document Auditing](https://mlsanity.com/paper/2608.16763/) (2026)

## Abstract (arXiv)

Understanding visually-rich business documents to extract structured data and automate business workflows has been receiving attention both in academia and industry. Although recent multi-modal language models have achieved impressive results, we find that existing benchmarks do not reflect the complexity of real documents seen in industry. In this work, we identify the desiderata for a more comprehensive benchmark and propose one we call Visually Rich Document Understanding (VRDU). VRDU contains two datasets that represent several challenges: rich schema including diverse data types as well as hierarchical entities, complex templates including tables and multi-column layouts, and diversity of different layouts (templates) within a single document type. We design few-shot and conventional experiment settings along with a carefully designed matching algorithm to evaluate extraction results. We report the performance of strong baselines and offer three observations: (1) generalizing to new document templates is still very challenging, (2) few-shot performance has a lot of headroom, and (3) models struggle with hierarchical fields such as line-items in an invoice. We plan to open source the benchmark and the evaluation toolkit. We hope this helps the community make progress on these challenging tasks in extracting structured data from visually rich documents.