# Evaluating historical text normalization systems: How well do they generalize?

arXiv: https://arxiv.org/abs/1804.02545 · HTML: https://mlsanity.com/paper/1804.02545/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2018.

## Built on by (most cited first, 11 of 11)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [ChroniclingAmericaQA: A Large-scale Question Answering Dataset based on Historical American Newspaper Pages](https://mlsanity.com/paper/2403.17859/) (2024)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluation](https://mlsanity.com/paper/2306.12916/) (2023)
- [Multilingual Event Extraction from Historical Newspaper Adverts](https://mlsanity.com/paper/2305.10928/) (2023)
- [PHD: Pixel-Based Language Modeling of Historical Documents](https://mlsanity.com/paper/2310.18343/) (2023)
- [Computational Morphology with Neural Network Approaches](https://mlsanity.com/paper/2105.09404/) (2021)
- [Recent advancements in computational morphology : A comprehensive survey](https://mlsanity.com/paper/2406.05424/) (2024)
- [Dealing with Abbreviations in the Slovenian Biographical Lexicon](https://mlsanity.com/paper/2211.02429/) (2022)
- [Two Approaches to Diachronic Normalization of Polish Texts](https://mlsanity.com/paper/2402.01300/) (2024)
- [Historical German Text Normalization Using Type- and Token-Based Language Modeling](https://mlsanity.com/paper/2409.02841/) (2024)

## Abstract (arXiv)

We highlight several issues in the evaluation of historical text normalization systems that make it hard to tell how well these systems would actually work in practice---i.e., for new datasets or languages; in comparison to more na"ive systems; or as a preprocessing step for downstream NLP tools. We illustrate these issues and exemplify our proposed evaluation practices by comparing two neural models against a na"ive baseline system. We show that the neural models generalize well to unseen words in tests on five languages; nevertheless, they provide no clear benefit over the na"ive baseline for downstream POS tagging of an English historical collection. We conclude that future work should include more rigorous evaluation, including both intrinsic and extrinsic measures where possible.