# DocILE Benchmark for Document Information Localization and Extraction

arXiv: https://arxiv.org/abs/2302.05658 · HTML: https://mlsanity.com/paper/2302.05658/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 107 references that are in the map.
- First posted on arXiv 11 Feb 2023.

## Builds on (most cited first, 30 of 67)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Real-time Scene Text Detection with Differentiable Binarization](https://mlsanity.com/paper/1911.08947/) (2019)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [PP-OCR: A Practical Ultra Lightweight OCR System](https://mlsanity.com/paper/2009.09941/) (2020)
- [VisualMRC: Machine Reading Comprehension on Document Images](https://mlsanity.com/paper/2101.11272/) (2021)
- [Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context](https://mlsanity.com/paper/2005.00589/) (2020)
- [Unifying Vision, Text, and Layout for Universal Document Processing](https://mlsanity.com/paper/2212.02623/) (2022)
- [ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019](https://mlsanity.com/paper/1907.00945/) (2019)
- [Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer](https://mlsanity.com/paper/2102.09550/) (2021)
- [PubTables-1M: Towards comprehensive table extraction from unstructured documents](https://mlsanity.com/paper/2110.00061/) (2021)
- [LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding](https://mlsanity.com/paper/2202.13669/) (2022)

## Built on by (most cited first, 30 of 58)

- [A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding](https://mlsanity.com/paper/2407.01976/) (2024)
- [DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming](https://mlsanity.com/paper/2406.19101/) (2024)
- [InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions](https://mlsanity.com/paper/2401.13313/) (2024)
- [DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding](https://mlsanity.com/paper/2408.15045/) (2024)
- [Deep Learning based Visually Rich Document Content Understanding: A Survey](https://mlsanity.com/paper/2408.01287/) (2024)
- [MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding](https://mlsanity.com/paper/2410.21311/) (2024)
- [Privacy-Aware Document Visual Question Answering](https://mlsanity.com/paper/2312.10108/) (2023)
- [Towards reducing hallucination in extracting information from financial reports using Large Language Models](https://mlsanity.com/paper/2310.10760/) (2023)
- [Beyond Document Page Classification: Design, Datasets, and Challenges](https://mlsanity.com/paper/2308.12896/) (2023)
- [ANLS* -- A Universal Document Processing Metric for Generative Large Language Models](https://mlsanity.com/paper/2402.03848/) (2024)
- [BuDDIE: A Business Document Dataset for Multi-task Information Extraction](https://mlsanity.com/paper/2404.04003/) (2024)
- [A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends](https://mlsanity.com/paper/2507.09861/) (2025)
- [Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Report](https://mlsanity.com/paper/2406.11403/) (2024)
- [Visual Template Inference for Data Extraction from Documents](https://mlsanity.com/paper/2501.06659/) (2025)
- [ProcTag: Process Tagging for Assessing the Efficacy of Document Instruction Data](https://mlsanity.com/paper/2407.12358/) (2024)
- [Document Intelligence in the Era of Large Language Models: A Survey](https://mlsanity.com/paper/2510.13366/) (2025)
- [UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents](https://mlsanity.com/paper/2602.07038/) (2026)
- [DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splitting](https://mlsanity.com/paper/2602.15958/) (2026)
- ["What is the value of {templates}?" Rethinking Document Information Extraction Datasets for LLMs](https://mlsanity.com/paper/2410.15484/) (2024)
- [Multi-Agent Interactive Question Generation Framework for Long Document Understanding](https://mlsanity.com/paper/2507.20145/) (2025)
- [Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligence](https://mlsanity.com/paper/2601.12318/) (2026)
- [VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents](https://mlsanity.com/paper/2603.15118/) (2026)
- [Compiled AI: Deterministic Code Generation for LLM-Based Workflow Automation](https://mlsanity.com/paper/2604.05150/) (2026)
- [Unveiling the Deficiencies of Pre-trained Text-and-Layout Models in Real-world Visually-rich Document Information Extraction](https://mlsanity.com/paper/2402.02379/) (2024)
- [TreeForm: End-to-end Annotation and Evaluation for Form Document Parsing](https://mlsanity.com/paper/2402.05282/) (2024)
- [Retrieval Augmented Structured Generation: Business Document Information Extraction As Tool Use](https://mlsanity.com/paper/2405.20245/) (2024)
- [RAPTOR: Refined Approach for Product Table Object Recognition](https://mlsanity.com/paper/2502.14918/) (2025)
- [PrIeD-KIE: Towards Privacy Preserved Document Key Information Extraction](https://mlsanity.com/paper/2310.03777/) (2023)
- [NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQA](https://mlsanity.com/paper/2411.03730/) (2024)
- [Training LayoutLM from Scratch for Efficient Named-Entity Recognition in the Insurance Domain](https://mlsanity.com/paper/2412.09341/) (2024)

## Abstract (arXiv)

This paper introduces the DocILE benchmark with the largest dataset of business documents for the tasks of Key Information Localization and Extraction and Line Item Recognition. It contains 6.7k annotated business documents, 100k synthetically generated documents, and nearly~1M unlabeled documents for unsupervised pre-training. The dataset has been built with knowledge of domain- and task-specific aspects, resulting in the following key features: (i) annotations in 55 classes, which surpasses the granularity of previously published key information extraction datasets by a large margin; (ii) Line Item Recognition represents a highly practical information extraction task, where key information has to be assigned to items in a table; (iii) documents come from numerous layouts and the test set includes zero- and few-shot cases as well as layouts commonly seen in the training set. The benchmark comes with several baselines, including RoBERTa, LayoutLMv3 and DETR-based Table Transformer; applied to both tasks of the DocILE benchmark, with results shared in this paper, offering a quick starting point for future work. The dataset, baselines and supplementary material are available at https://github.com/rossumai/docile.