# Learning to Align: Addressing Character Frequency Distribution Shifts in Handwritten Text Recognition

arXiv: https://arxiv.org/abs/2506.09846 · HTML: https://mlsanity.com/paper/2506.09846/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 48 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints](https://mlsanity.com/paper/2010.12884/) (2020)
- [Guided Open Vocabulary Image Captioning with Constrained Beam Search](https://mlsanity.com/paper/1612.00576/) (2016)
- [Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP](https://mlsanity.com/paper/2208.05516/) (2022)
- [Unsupervised Text Style Transfer using Language Models as Discriminators](https://mlsanity.com/paper/1805.11749/) (2018)
- [COLLIE: Systematic Construction of Constrained Text Generation Tasks](https://mlsanity.com/paper/2307.08689/) (2023)
- [A Plug-and-Play Method for Controlled Text Generation](https://mlsanity.com/paper/2109.09707/) (2021)
- [The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR](https://mlsanity.com/paper/2303.18110/) (2023)
- [Boosting Modern and Historical Handwritten Text Recognition with Deformable Convolutions](https://mlsanity.com/paper/2208.08109/) (2022)
- [Cross-domain Speech Recognition with Unsupervised Character-level Distribution Matching](https://mlsanity.com/paper/2104.07491/) (2021)
- [Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio](https://mlsanity.com/paper/2306.15926/) (2023)
- [Removing Distributional Discrepancies in Captions Improves Image-Text Alignment](https://mlsanity.com/paper/2410.00905/) (2024)
- [A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning](https://mlsanity.com/paper/2406.14164/) (2024)
- [An Analysis of Letter Dynamics in the English Alphabet](https://mlsanity.com/paper/2401.15560/) (2024)

## Built on by (most cited first, 1 of 1)

- [FineFreq: A Multilingual Character Frequency Dataset from Web-Scale Text](https://mlsanity.com/paper/2512.09701/) (2025)

## Abstract (arXiv)

Handwritten text recognition aims to convert visual input into machine-readable text, and it remains challenging due to the evolving and context-dependent nature of handwriting. Character sets change over time, and character frequency distributions shift across historical periods or regions, often causing models trained on broad, heterogeneous corpora to underperform on specific subsets. To tackle this, we propose a novel loss function that incorporates the Wasserstein distance between the character frequency distribution of the predicted text and a target distribution empirically derived from training data. By penalizing divergence from expected distributions, our approach enhances both accuracy and robustness under temporal and contextual intra-dataset shifts. Furthermore, we demonstrate that character distribution alignment can also improve existing models at inference time without requiring retraining by integrating it as a scoring function in a guided decoding scheme. Experimental results across multiple datasets and architectures confirm the effectiveness of our method in boosting generalization and performance. We open source our code at https://github.com/pkaliosis/fada.