# Latin BERT: A Contextual Language Model for Classical Philology

arXiv: https://arxiv.org/abs/2009.10053 · HTML: https://mlsanity.com/paper/2009.10053/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2020.

## Built on by (most cited first, 30 of 43)

- [Named Entity Recognition and Classification on Historical Documents: A Survey](https://mlsanity.com/paper/2109.11406/) (2021)
- [A Survey on Contextualised Semantic Shift Detection](https://mlsanity.com/paper/2304.01666/) (2023)
- [One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks](https://mlsanity.com/paper/2409.13920/) (2024)
- [Statistically significant detection of semantic shifts using contextual word embeddings](https://mlsanity.com/paper/2104.03776/) (2021)
- [A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Models](https://mlsanity.com/paper/2201.05601/) (2022)
- [Exploring Large Language Models for Classical Philology](https://mlsanity.com/paper/2305.13698/) (2023)
- [Sui Generis: Large Language Models for Authorship Attribution and Verification in Latin](https://mlsanity.com/paper/2410.09245/) (2024)
- [Introducing Rhetorical Parallelism Detection: A New Task with Datasets, Metrics, and Baselines](https://mlsanity.com/paper/2312.00100/) (2023)
- [BERT in Plutarch's Shadows](https://mlsanity.com/paper/2211.05673/) (2022)
- [LogogramNLP: Comparing Visual and Textual Representations of Ancient Logographic Writing Systems for NLP](https://mlsanity.com/paper/2408.04628/) (2024)
- [HUE: Pretrained Model and Dataset for Understanding Hanja Documents of Ancient Korea](https://mlsanity.com/paper/2210.05112/) (2022)
- [LatinCy: Synthetic Trained Pipelines for Latin NLP](https://mlsanity.com/paper/2305.04365/) (2023)
- [From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French](https://mlsanity.com/paper/2202.09452/) (2022)
- [Graecia capta ferum victorem cepit. Detecting Latin Allusions to Ancient Greek Literature](https://mlsanity.com/paper/2308.12008/) (2023)
- [Reviving Cultural Heritage: A Novel Approach for Comprehensive Historical Document Restoration](https://mlsanity.com/paper/2507.05108/) (2025)
- [Insightful analysis of historical sources at scales beyond human capabilities using unsupervised Machine Learning and XAI](https://mlsanity.com/paper/2310.09091/) (2023)
- [Surveying the Dead Minds: Historical-Psychological Text Analysis with Contextualized Construct Representation (CCR) for Classical Chinese](https://mlsanity.com/paper/2403.00509/) (2024)
- [Instruct-Tuning Pretrained Causal Language Models for Ancient Greek Papyrology and Epigraphy](https://mlsanity.com/paper/2409.13870/) (2024)
- [Loci Similes: A Benchmark for Extracting Intertextualities in Latin Literature](https://mlsanity.com/paper/2601.07533/) (2026)
- ['UFAL LatinPipe at EvaLatin 2024: Morphosyntactic Analysis of Latin](https://mlsanity.com/paper/2404.05839/) (2024)
- [PaliBench: A Multi-Reference Blueprint for Classical Language Translation Benchmarks](https://mlsanity.com/paper/2605.16881/) (2026)
- [Latin writing styles analysis with Machine Learning: New approach to old questions](https://mlsanity.com/paper/2109.00601/) (2021)
- [Metadata Might Make Language Models Better](https://mlsanity.com/paper/2211.10086/) (2022)
- [Contrastive Attention Networks for Attribution of Early Modern Print](https://mlsanity.com/paper/2306.07998/) (2023)
- [Detecting Sexual Content at the Sentence Level in First Millennium Latin Texts](https://mlsanity.com/paper/2309.14974/) (2023)
- [Larth: Dataset and Machine Translation for Etruscan](https://mlsanity.com/paper/2310.05688/) (2023)
- [Heidelberg-Boston @ SIGTYP 2024 Shared Task: Enhancing Low-Resource Language Analysis With Character-Aware Hierarchical Transformers](https://mlsanity.com/paper/2405.20145/) (2024)
- [Latin Treebanks in Review: An Evaluation of Morphological Tagging Across Time](https://mlsanity.com/paper/2408.06675/) (2024)
- [Comparative Analysis of Static and Contextual Embeddings for Analyzing Semantic Changes in Medieval Latin Charters](https://mlsanity.com/paper/2410.09283/) (2024)
- [Contextual morphologically-guided tokenization for Latin encoder models](https://mlsanity.com/paper/2511.09709/) (2025)

## Abstract (arXiv)

We present Latin BERT, a contextual language model for the Latin language, trained on 642.7 million words from a variety of sources spanning the Classical era to the 21st century. In a series of case studies, we illustrate the affordances of this language-specific model both for work in natural language processing for Latin and in using computational methods for traditional scholarship: we show that Latin BERT achieves a new state of the art for part-of-speech tagging on all three Universal Dependency datasets for Latin and can be used for predicting missing text (including critical emendations); we create a new dataset for assessing word sense disambiguation for Latin and demonstrate that Latin BERT outperforms static word embeddings; and we show that it can be used for semantically-informed search by querying contextual nearest neighbors. We publicly release trained models to help drive future work in this space.