# Mixed Model OCR Training on Historical Latin Script for Out-of-the-Box Recognition and Finetuning

arXiv: https://arxiv.org/abs/2106.07881 · HTML: https://mlsanity.com/paper/2106.07881/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 23 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 10 of 10)

- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [A selectional auto-encoder approach for document image binarization](https://mlsanity.com/paper/1706.10241/) (2017)
- [Calamari - A High-Performance Tensorflow-based Deep Learning Package for Optical Character Recognition](https://mlsanity.com/paper/1807.02004/) (2018)
- [Ground Truth for training OCR engines on historical documents in German Fraktur and Early Modern Latin](https://mlsanity.com/paper/1809.05501/) (2018)
- [OCR4all -- An Open-Source Tool Providing a (Semi-)Automatic OCR Workflow for Historical Printings](https://mlsanity.com/paper/1909.04032/) (2019)
- [OCR of historical printings with an application to building diachronic corpora: A case study using the RIDGES herbal corpus](https://mlsanity.com/paper/1608.02153/) (2016)
- [Improving OCR Accuracy on Early Printed Books by utilizing Cross Fold Training and Voting](https://mlsanity.com/paper/1711.09670/) (2017)
- [State of the Art Optical Character Recognition of 19th Century Fraktur Scripts using Open Source Engines](https://mlsanity.com/paper/1810.03436/) (2018)
- [Improving OCR Accuracy on Early Printed Books by combining Pretraining, Voting, and Active Learning](https://mlsanity.com/paper/1802.10038/) (2018)
- [Automatic quality evaluation and (semi-) automatic improvement of OCR models for historical printings](https://mlsanity.com/paper/1606.05157/) (2016)

## Built on by (most cited first, 6 of 6)

- [Multimodal LLMs for OCR, OCR Post-Correction, and Named Entity Recognition in Historical Documents](https://mlsanity.com/paper/2504.00414/) (2025)
- [Towards Writing Style Adaptation in Handwriting Recognition](https://mlsanity.com/paper/2302.06318/) (2023)
- [Ancient Script Image Recognition and Processing: A Review](https://mlsanity.com/paper/2506.19208/) (2025)
- [Fine-tuning Is a Surprisingly Effective Domain Adaptation Baseline in Handwriting Recognition](https://mlsanity.com/paper/2302.06308/) (2023)
- [Open Source Handwritten Text Recognition on Medieval Manuscripts using Mixed Models and Document-Specific Finetuning](https://mlsanity.com/paper/2201.07661/) (2022)
- [Practical Fine-Tuning of Autoregressive Models on Limited Handwritten Texts](https://mlsanity.com/paper/2503.19546/) (2025)

## Abstract (arXiv)

In order to apply Optical Character Recognition (OCR) to historical printings of Latin script fully automatically, we report on our efforts to construct a widely-applicable polyfont recognition model yielding text with a Character Error Rate (CER) around 2% when applied out-of-the-box. Moreover, we show how this model can be further finetuned to specific classes of printings with little manual and computational effort. The mixed or polyfont model is trained on a wide variety of materials, in terms of age (from the 15th to the 19th century), typography (various types of Fraktur and Antiqua), and languages (among others, German, Latin, and French). To optimize the results we combined established techniques of OCR training like pretraining, data augmentation, and voting. In addition, we used various preprocessing methods to enrich the training data and obtain more robust models. We also implemented a two-stage approach which first trains on all available, considerably unbalanced data and then refines the output by training on a selected more balanced subset. Evaluations on 29 previously unseen books resulted in a CER of 1.73%, outperforming a widely used standard model with a CER of 2.84% by almost 40%. Training a more specialized model for some unseen Early Modern Latin books starting from our mixed model led to a CER of 1.47%, an improvement of up to 50% compared to training from scratch and up to 30% compared to training from the aforementioned standard model. Our new mixed model is made openly available to the community.