# Evaluation of HTR models without Ground Truth Material

arXiv: https://arxiv.org/abs/2201.06170 · HTML: https://mlsanity.com/paper/2201.06170/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 39 references that are in the map.
- First posted on arXiv 17 Jan 2022.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Candidate Fusion: Integrating Language Modelling into a Sequence-to-Sequence Handwritten Word Recognition Architecture](https://mlsanity.com/paper/1912.10308/) (2019)
- [Implicit Language Model in LSTM for OCR](https://mlsanity.com/paper/1805.09441/) (2018)

## Built on by (most cited first, 5 of 5)

- [Benchmarking Large Language Models for Handwritten Text Recognition](https://mlsanity.com/paper/2503.15195/) (2025)
- [End-to-End Page-Level Assessment of Handwritten Text Recognition](https://mlsanity.com/paper/2301.05935/) (2023)
- [Layout-Aware OCR for Black Digital Archives with Unsupervised Evaluation](https://mlsanity.com/paper/2509.13236/) (2025)
- [The Challenges of HTR Model Training: Feedback from the Project Donner le gout de l'archive a l'ere numerique](https://mlsanity.com/paper/2212.11146/) (2022)
- [A machine-readable catalogue of the Tsiolkovsky papers (fond 555, Archive of the Russian Academy of Sciences), and a way to measure how well its handwriting can be read](https://mlsanity.com/paper/2608.03617/) (2026)

## Abstract (arXiv)

The evaluation of Handwritten Text Recognition (HTR) models during their development is straightforward: because HTR is a supervised problem, the usual data split into training, validation, and test data sets allows the evaluation of models in terms of accuracy or error rates. However, the evaluation process becomes tricky as soon as we switch from development to application. A compilation of a new (and forcibly smaller) ground truth (GT) from a sample of the data that we want to apply the model on and the subsequent evaluation of models thereon only provides hints about the quality of the recognised text, as do confidence scores (if available) the models return. Moreover, if we have several models at hand, we face a model selection problem since we want to obtain the best possible result during the application phase. This calls for GT-free metrics to select the best model, which is why we (re-)introduce and compare different metrics, from simple, lexicon-based to more elaborate ones using standard language models and masked language models (MLM). We show that MLM-based evaluation can compete with lexicon-based methods, with the advantage that large and multilingual transformers are readily available, thus making compiling lexical resources for other metrics superfluous.