# hmBERT: Historical Multilingual Language Models for Named Entity Recognition

arXiv: https://arxiv.org/abs/2205.15575 · HTML: https://mlsanity.com/paper/2205.15575/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 31 May 2022.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018)
- [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://mlsanity.com/paper/2012.15562/) (2020)
- [Automated Concatenation of Embeddings for Structured Prediction](https://mlsanity.com/paper/2010.05006/) (2020)
- [FLERT: Document-Level Features for Named Entity Recognition](https://mlsanity.com/paper/2011.06993/) (2020)
- [Token Dropping for Efficient BERT Pretraining](https://mlsanity.com/paper/2203.13240/) (2022)
- [Interpretable Multi-dataset Evaluation for Named Entity Recognition](https://mlsanity.com/paper/2011.06854/) (2020)
- [DAMO-NLP at SemEval-2022 Task 11: A Knowledge-based System for Multilingual Named Entity Recognition](https://mlsanity.com/paper/2203.00545/) (2022)
- [Neural Language Models for Nineteenth-Century English](https://mlsanity.com/paper/2105.11321/) (2021)

## Built on by (most cited first, 8 of 8)

- [Yes but.. Can ChatGPT Identify Entities in Historical Documents?](https://mlsanity.com/paper/2303.17322/) (2023)
- [SwissBERT: The Multilingual Language Model for Switzerland](https://mlsanity.com/paper/2303.13310/) (2023)
- [A Survey of Pre-trained Language Models for Processing Scientific Text](https://mlsanity.com/paper/2401.17824/) (2024)
- [Metadata Might Make Language Models Better](https://mlsanity.com/paper/2211.10086/) (2022)
- [Transfer Learning across Several Centuries: Machine and Historian Integrated Method to Decipher Royal Secretary's Diary](https://mlsanity.com/paper/2306.14592/) (2023)
- [Investigating OCR-Sensitive Neurons to Improve Entity Recognition in Historical Documents](https://mlsanity.com/paper/2409.16934/) (2024)
- [Named Entity Recognition of Historical Texts via Large Language Model](https://mlsanity.com/paper/2508.18090/) (2025)
- [A Study of Temporal Fusion Strategies for Named Entity Recognition in Historical Texts](https://mlsanity.com/paper/2606.27881/) (2026)

## Abstract (arXiv)

Compared to standard Named Entity Recognition (NER), identifying persons, locations, and organizations in historical texts constitutes a big challenge. To obtain machine-readable corpora, the historical text is usually scanned and Optical Character Recognition (OCR) needs to be performed. As a result, the historical corpora contain errors. Also, entities like location or organization can change over time, which poses another challenge. Overall, historical texts come with several peculiarities that differ greatly from modern texts and large labeled corpora for training a neural tagger are hardly available for this domain. In this work, we tackle NER for historical German, English, French, Swedish, and Finnish by training large historical language models. We circumvent the need for large amounts of labeled data by using unlabeled data for pretraining a language model. We propose hmBERT, a historical multilingual BERT-based language model, and release the model in several versions of different sizes. Furthermore, we evaluate the capability of hmBERT by solving downstream NER as part of this year's HIPE-2022 shared task and provide detailed analysis and insights. For the Multilingual Classical Commentary coarse-grained NER challenge, our tagger HISTeria outperforms the other teams' models for two out of three languages.