# Lacuna Reconstruction: Self-supervised Pre-training for Low-Resource Historical Document Transcription

arXiv: https://arxiv.org/abs/2112.08692 · HTML: https://mlsanity.com/paper/2112.08692/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Effectiveness of self-supervised pre-training for speech recognition](https://mlsanity.com/paper/1911.03912/) (2019)
- [Pre-Training Transformers as Energy-Based Cloze Models](https://mlsanity.com/paper/2012.08561/) (2020)
- [Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstruction](https://mlsanity.com/paper/2001.10603/) (2020)
- [OCR Post Correction for Endangered Language Texts](https://mlsanity.com/paper/2011.05402/) (2020)
- [Improving Transformer-based Speech Recognition Using Unsupervised Pre-training](https://mlsanity.com/paper/1910.09932/) (2019)
- [A Scalable Handwritten Text Recognition System](https://mlsanity.com/paper/1904.09150/) (2019)
- [Joint Masked CPC and CTC Training for ASR](https://mlsanity.com/paper/2011.00093/) (2020)
- [Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks](https://mlsanity.com/paper/1910.10387/) (2019)
- [Sequence-to-Label Script Identification for Multilingual OCR](https://mlsanity.com/paper/1708.04671/) (2017)
- [Lexically Aware Semi-Supervised Learning for OCR Post-Correction](https://mlsanity.com/paper/2111.02622/) (2021)
- [Boosting Handwriting Text Recognition in Small Databases with Transfer Learning](https://mlsanity.com/paper/1804.01527/) (2018)
- [From the Paft to the Fiiture: a Fully Automatic NMT and Word Embeddings Method for OCR Post-Correction](https://mlsanity.com/paper/1910.05535/) (2019)
- [Important New Developments in Arabographic Optical Character Recognition (OCR)](https://mlsanity.com/paper/1703.09550/) (2017)
- [A Probabilistic Generative Model for Typographical Analysis of Early Modern Printing](https://mlsanity.com/paper/2005.01646/) (2020)

## Built on by (most cited first, 3 of 3)

- [Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and a Vision-Language Model](https://mlsanity.com/paper/2602.14524/) (2026)
- [Contrastive Attention Networks for Attribution of Early Modern Print](https://mlsanity.com/paper/2306.07998/) (2023)
- [Lacuna Language Learning: Leveraging RNNs for Ranked Text Completion in Digitized Coptic Manuscripts](https://mlsanity.com/paper/2407.12247/) (2024)

## Abstract (arXiv)

We present a self-supervised pre-training approach for learning rich visual language representations for both handwritten and printed historical document transcription. After supervised fine-tuning of our pre-trained encoder representations for low-resource document transcription on two languages, (1) a heterogeneous set of handwritten Islamicate manuscript images and (2) early modern English printed documents, we show a meaningful improvement in recognition accuracy over the same supervised model trained from scratch with as few as 30 line image transcriptions for training. Our masked language model-style pre-training strategy, where the model is trained to be able to identify the true masked visual representation from distractors sampled from within the same line, encourages learning robust contextualized language representations invariant to scribal writing style and printing noise present across documents.