# Historical German Text Normalization Using Type- and Token-Based Language Modeling

arXiv: https://arxiv.org/abs/2409.02841 · HTML: https://mlsanity.com/paper/2409.02841/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 54 references that are in the map.
- First posted on arXiv 4 Sep 2024.

## Builds on (most cited first, 10 of 10)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Applying the Transformer to Character-level Transduction](https://mlsanity.com/paper/2005.10213/) (2020)
- [A Large-Scale Comparison of Historical Text Normalization Systems](https://mlsanity.com/paper/1904.02036/) (2019)
- [Evaluating historical text normalization systems: How well do they generalize?](https://mlsanity.com/paper/1804.02545/) (2018)
- [Sequence-to-Sequence Lexical Normalization with Multilingual Transformers](https://mlsanity.com/paper/2110.02869/) (2021)
- [An Evaluation of Neural Machine Translation Models on Historical Spelling Normalization](https://mlsanity.com/paper/1806.05210/) (2018)
- [Modeling Orthographic Variation in Occitan's Dialects](https://mlsanity.com/paper/2404.19315/) (2024)

## Abstract (arXiv)

Historic variations of spelling poses a challenge for full-text search or natural language processing on historical digitized texts. To minimize the gap between the historic orthography and contemporary spelling, usually an automatic orthographic normalization of the historical source material is pursued. This report proposes a normalization system for German literary texts from c. 1700-1900, trained on a parallel corpus. The proposed system makes use of a machine learning approach using Transformer language models, combining an encoder-decoder model to normalize individual word types, and a pre-trained causal language model to adjust these normalizations within their context. An extensive evaluation shows that the proposed system provides state-of-the-art accuracy, comparable with a much larger fully end-to-end sentence-based normalization system, fine-tuning a pre-trained Transformer large language model. However, the normalization of historical text remains a challenge due to difficulties for models to generalize, and the lack of extensive high-quality parallel data.