# OCR-IDL: OCR Annotations for Industry Document Library Dataset

arXiv: https://arxiv.org/abs/2202.12985 · HTML: https://mlsanity.com/paper/2202.12985/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 54 references that are in the map.
- First posted on arXiv 25 Feb 2022.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [InfographicVQA](https://mlsanity.com/paper/2104.12756/) (2021)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [DocBank: A Benchmark Dataset for Document Layout Analysis](https://mlsanity.com/paper/2006.01038/) (2020)
- [Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer](https://mlsanity.com/paper/2102.09550/) (2021)
- [SelfDoc: Self-Supervised Document Representation Learning](https://mlsanity.com/paper/2106.03331/) (2021)
- [CascadeTabNet: An approach for end to end table detection and structure recognition from image-based documents](https://mlsanity.com/paper/2004.12629/) (2020)
- [Variational Transformer Networks for Layout Generation](https://mlsanity.com/paper/2104.02416/) (2021)
- [LaTr: Layout-Aware Transformer for Scene-Text VQA](https://mlsanity.com/paper/2112.12494/) (2021)
- [Sequence-to-Sequence Contrastive Learning for Text Recognition](https://mlsanity.com/paper/2012.10873/) (2020)
- [SCATTER: Selective Context Attentional Scene Text Recognizer](https://mlsanity.com/paper/2003.11288/) (2020)
- [Table Structure Recognition using Top-Down and Bottom-Up Cues](https://mlsanity.com/paper/2010.04565/) (2020)
- [READ: Recursive Autoencoders for Document Layout Generation](https://mlsanity.com/paper/1909.00302/) (2019)
- [ICDAR 2019 Competition on Scene Text Visual Question Answering](https://mlsanity.com/paper/1907.00490/) (2019)
- [Document Collection Visual Question Answering](https://mlsanity.com/paper/2104.14336/) (2021)
- [DE-GAN: A Conditional Generative Adversarial Network for Document Enhancement](https://mlsanity.com/paper/2010.08764/) (2020)
- [Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Features](https://mlsanity.com/paper/2111.15263/) (2021)

## Built on by (most cited first, 30 of 35)

- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [Building and better understanding vision-language models: insights and future directions](https://mlsanity.com/paper/2408.12637/) (2024)
- [xGen-MM (BLIP-3): A Family of Open Large Multimodal Models](https://mlsanity.com/paper/2408.08872/) (2024)
- [Hierarchical multimodal transformers for Multi-Page DocVQA](https://mlsanity.com/paper/2212.05935/) (2022)
- [Detect Anything via Next Point Prediction](https://mlsanity.com/paper/2510.12798/) (2025)
- [DocILE Benchmark for Document Information Localization and Extraction](https://mlsanity.com/paper/2302.05658/) (2023)
- [Scalable Vision Language Model Training via High Quality Data Curation](https://mlsanity.com/paper/2501.05952/) (2025)
- [SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion](https://mlsanity.com/paper/2503.11576/) (2025)
- [LongCat-Next: Lexicalizing Modalities as Discrete Tokens](https://mlsanity.com/paper/2603.27538/) (2026)
- [DocVLM: Make Your VLM an Efficient Reader](https://mlsanity.com/paper/2412.08746/) (2024)
- [Scaling Vision Pre-Training to 4K Resolution](https://mlsanity.com/paper/2503.19903/) (2025)
- [CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Data](https://mlsanity.com/paper/2304.14953/) (2023)
- [M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance](https://mlsanity.com/paper/2502.18778/) (2025)
- [Beyond Document Page Classification: Design, Datasets, and Challenges](https://mlsanity.com/paper/2308.12896/) (2023)
- [Towards Understanding Visual Grounding in Visual Language Models](https://mlsanity.com/paper/2509.10345/) (2025)
- [VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document Understanding](https://mlsanity.com/paper/2407.12594/) (2024)
- [Graph Neural Networks and Representation Embedding for Table Extraction in PDF Documents](https://mlsanity.com/paper/2208.11203/) (2022)
- [On Web-based Visual Corpus Construction for Visual Document Understanding](https://mlsanity.com/paper/2211.03256/) (2022)
- [On Evaluation of Document Classification using RVL-CDIP](https://mlsanity.com/paper/2306.12550/) (2023)
- [Large Language Models for Page Stream Segmentation](https://mlsanity.com/paper/2408.11981/) (2024)
- [Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark](https://mlsanity.com/paper/2512.20174/) (2025)
- [MUST-VQA: MUltilingual Scene-text VQA](https://mlsanity.com/paper/2209.06730/) (2022)
- [SynthDoc: Bilingual Documents Synthesis for Visual Document Understanding](https://mlsanity.com/paper/2408.14764/) (2024)
- [SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinement](https://mlsanity.com/paper/2507.01643/) (2025)
- [Context-Aware Classification of Legal Document Pages](https://mlsanity.com/paper/2304.02787/) (2023)
- [AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretraining](https://mlsanity.com/paper/2506.13274/) (2025)
- [Can AI Models Appreciate Document Aesthetics? An Exploration of Legibility and Layout Quality in Relation to Prediction Confidence](https://mlsanity.com/paper/2403.18183/) (2024)
- [TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models](https://mlsanity.com/paper/2411.04642/) (2024)
- [Towards Multi-Modal Mastery: A 4.5B Parameter Truly Multi-Modal Small Language Model](https://mlsanity.com/paper/2411.05903/) (2024)

## Abstract (arXiv)

Pretraining has proven successful in Document Intelligence tasks where deluge of documents are used to pretrain the models only later to be finetuned on downstream tasks. One of the problems of the pretraining approaches is the inconsistent usage of pretraining data with different OCR engines leading to incomparable results between models. In other words, it is not obvious whether the performance gain is coming from diverse usage of amount of data and distinct OCR engines or from the proposed models. To remedy the problem, we make public the OCR annotations for IDL documents using commercial OCR engine given their superior performance over open source OCR models. The contributed dataset (OCR-IDL) has an estimated monetary value over 20K US. It is our hope that OCR-IDL can be a starting point for future works on Document Intelligence. All of our data and its collection process with the annotations can be found in https://github.com/furkanbiten/idldata.