# On the Generalization of Handwritten Text Recognition Models

arXiv: https://arxiv.org/abs/2411.17332 · HTML: https://mlsanity.com/paper/2411.17332/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 97 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 30 of 60)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [A Survey on Deep Transfer Learning](https://mlsanity.com/paper/1808.01974/) (2018)
- [A Survey of Unsupervised Deep Domain Adaptation](https://mlsanity.com/paper/1812.02849/) (2018)
- [Learning to Learn Single Domain Generalization](https://mlsanity.com/paper/2003.13216/) (2020)
- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [Improving Unsupervised Defect Segmentation by Applying Structural Similarity to Autoencoders](https://mlsanity.com/paper/1807.02011/) (2018)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization](https://mlsanity.com/paper/2107.04649/) (2021)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [Learning to Diversify for Single Domain Generalization](https://mlsanity.com/paper/2108.11726/) (2021)
- [Domino: Discovering Systematic Errors with Cross-Modal Embeddings](https://mlsanity.com/paper/2203.14960/) (2022)
- [Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer](https://mlsanity.com/paper/2108.03032/) (2021)
- [Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift](https://mlsanity.com/paper/2206.13089/) (2022)
- [Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognition](https://mlsanity.com/paper/2005.13044/) (2020)
- [Sequence-to-Sequence Contrastive Learning for Text Recognition](https://mlsanity.com/paper/2012.10873/) (2020)
- [DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition](https://mlsanity.com/paper/2203.12273/) (2022)
- [Evaluating Sequence-to-Sequence Models for Handwritten Text Recognition](https://mlsanity.com/paper/1903.07377/) (2019)

## Built on by (most cited first, 10 of 10)

- [Handwritten Text Recognition: A Survey](https://mlsanity.com/paper/2502.08417/) (2025)
- [Few-shot Writer Adaptation via Multimodal In-Context Learning](https://mlsanity.com/paper/2603.29450/) (2026)
- [N-gram Injection into Transformers for Dynamic Language Model Adaptation in Handwritten Text Recognition](https://mlsanity.com/paper/2603.03930/) (2026)
- [Zero-Shot Synthetic-to-Real Handwritten Text Recognition via Task Analogies](https://mlsanity.com/paper/2604.09713/) (2026)
- [Out-of-Length Scene Text Recognition: A Two-Axis Diagnosis and a Training-Free Fix](https://mlsanity.com/paper/2607.23194/) (2026)
- [UniLipi: A Unified Multi-Script OCR for Historical Indic Manuscripts](https://mlsanity.com/paper/2608.28195/) (2026)
- [ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts](https://mlsanity.com/paper/2609.12705/) (2026)
- [A Free Lunch? Adapting PP-OCRv6 for Historical Text Recognition](https://mlsanity.com/paper/2609.20064/) (2026)
- [Handwritten Text Recognition Lives in the High-Pixel Variance Subspace](https://mlsanity.com/paper/2609.35473/) (2026)
- [Exploring In-Context Learning for Handwritten Text Recognition](https://mlsanity.com/paper/2609.37195/) (2026)

## Abstract (arXiv)

Recent advances in Handwritten Text Recognition (HTR) have led to significant reductions in transcription errors on standard benchmarks under the i.i.d. assumption, thus focusing on minimizing in-distribution (ID) errors. However, this assumption does not hold in real-world applications, which has motivated HTR research to explore Transfer Learning and Domain Adaptation techniques. In this work, we investigate the unaddressed limitations of HTR models in generalizing to out-of-distribution (OOD) data. We adopt the challenging setting of Domain Generalization, where models are expected to generalize to OOD data without any prior access. To this end, we analyze 336 OOD cases from eight state-of-the-art HTR models across seven widely used datasets, spanning five languages. Additionally, we study how HTR models leverage synthetic data to generalize. We reveal that the most significant factor for generalization lies in the textual divergence between domains, followed by visual divergence. We demonstrate that the error of HTR models in OOD scenarios can be reliably estimated, with discrepancies falling below 10 points in 70% of cases. We identify the underlying limitations of HTR models, laying the foundation for future research to address this challenge.