# A Survey of Historical Document Image Datasets

arXiv: https://arxiv.org/abs/2203.08504 · HTML: https://mlsanity.com/paper/2203.08504/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 202 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [Robust Classification with Convolutional Prototype Learning](https://mlsanity.com/paper/1805.03438/) (2018)
- [CascadeTabNet: An approach for end to end table detection and structure recognition from image-based documents](https://mlsanity.com/paper/2004.12629/) (2020)
- [End-to-end Handwritten Paragraph Text Recognition Using a Vertical Attention Network](https://mlsanity.com/paper/2012.03868/) (2020)
- [dhSegment: A generic deep-learning approach for document segmentation](https://mlsanity.com/paper/1804.10371/) (2018)
- [LadderNet: Multi-path networks based on U-Net for medical image segmentation](https://mlsanity.com/paper/1810.07810/) (2018)
- [A Large Dataset of Historical Japanese Documents with Complex Layouts](https://mlsanity.com/paper/2004.08686/) (2020)
- [Unsupervised Feature Learning for Writer Identification and Writer Retrieval](https://mlsanity.com/paper/1705.09369/) (2017)

## Built on by (most cited first, 20 of 20)

- [WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models](https://mlsanity.com/paper/2303.16576/) (2023)
- [Around the World in 80 Timesteps: A Generative Approach to Global Visual Geolocation](https://mlsanity.com/paper/2412.06781/) (2024)
- [Efficient OCR for Building a Diverse Digital History](https://mlsanity.com/paper/2304.02737/) (2023)
- [Rethinking HTG Evaluation: Bridging Generation and Recognition](https://mlsanity.com/paper/2409.02683/) (2024)
- [U-DIADS-Bib: a full and few-shot pixel-precise dataset for document layout analysis of ancient manuscripts](https://mlsanity.com/paper/2401.08425/) (2024)
- [Insightful analysis of historical sources at scales beyond human capabilities using unsupervised Machine Learning and XAI](https://mlsanity.com/paper/2310.09091/) (2023)
- [Self-Supervised Learning for Text Recognition: A Critical Survey](https://mlsanity.com/paper/2407.19889/) (2024)
- [Fetch-A-Set: A Large-Scale OCR-Free Benchmark for Historical Document Retrieval](https://mlsanity.com/paper/2406.07315/) (2024)
- [Quo Vadis Handwritten Text Generation for Handwritten Text Recognition?](https://mlsanity.com/paper/2508.09936/) (2025)
- [Dual Orthogonal Guidance for Robust Diffusion-based Handwritten Text Generation](https://mlsanity.com/paper/2508.17017/) (2025)
- [Prompt me a Dataset: An investigation of text-image prompting for historical image dataset creation using foundation models](https://mlsanity.com/paper/2309.01674/) (2023)
- [The OCR Quest for Generalization: Learning to recognize low-resource alphabets with model editing](https://mlsanity.com/paper/2506.06761/) (2025)
- [Multi-Expert Routing for Multi-Domain Low-Resource OCR: A Manchu Case Study](https://mlsanity.com/paper/2607.14041/) (2026)
- [SIMARA: a database for key-value information extraction from full pages](https://mlsanity.com/paper/2304.13606/) (2023)
- [CENSUS-HWR: a large training dataset for offline handwriting recognition](https://mlsanity.com/paper/2305.16275/) (2023)
- [Page image classifier fine-tuned on century-spanning archives of scanned documents for further content-specific processing](https://mlsanity.com/paper/2606.07558/) (2026)
- [Systematic Literature Review of Machine Learning Models and Applications for Text Recognition](https://mlsanity.com/paper/2608.26500/) (2026)
- [PalmLeaf-VQA: A Multi-Script Visual Question Answering Benchmark for Historical Palm-Leaf Manuscript Understanding Across Diverse Regions](https://mlsanity.com/paper/2609.31651/) (2026)
- [Exploring In-Context Learning for Handwritten Text Recognition](https://mlsanity.com/paper/2609.37195/) (2026)
- [DDVR-Net: Degradation-Aware Decoupling and Visibility-Guided Restoration for Historical Document Recognition](https://mlsanity.com/paper/2610.16956/) (2026)

## Abstract (arXiv)

This paper presents a systematic literature review of image datasets for document image analysis, focusing on historical documents, such as handwritten manuscripts and early prints. Finding appropriate datasets for historical document analysis is a crucial prerequisite to facilitate research using different machine learning algorithms. However, because of the very large variety of the actual data (e.g., scripts, tasks, dates, support systems, and amount of deterioration), the different formats for data and label representation, and the different evaluation processes and benchmarks, finding appropriate datasets is a difficult task. This work fills this gap, presenting a meta-study on existing datasets. After a systematic selection process (according to PRISMA guidelines), we select 65 studies that are chosen based on different factors, such as the year of publication, number of methods implemented in the article, reliability of the chosen algorithms, dataset size, and journal outlet. We summarize each study by assigning it to one of three pre-defined tasks: document classification, layout structure, or content analysis. We present the statistics, document type, language, tasks, input visual aspects, and ground truth information for every dataset. In addition, we provide the benchmark tasks and results from these papers or recent competitions. We further discuss gaps and challenges in this domain. We advocate for providing conversion tools to common formats (e.g., COCO format for computer vision tasks) and always providing a set of evaluation metrics, instead of just one, to make results comparable across studies.