# Contextual morphologically-guided tokenization for Latin encoder models

arXiv: https://arxiv.org/abs/2511.09709 · HTML: https://mlsanity.com/paper/2511.09709/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 61 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Builds on (most cited first, 23 of 23)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://mlsanity.com/paper/2007.11464/) (2020)
- [Tokenization Is More Than Compression](https://mlsanity.com/paper/2402.18376/) (2024)
- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://mlsanity.com/paper/2101.00403/) (2021)
- [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022)
- [Why do language models perform worse for morphologically complex languages?](https://mlsanity.com/paper/2411.14198/) (2024)
- [Latin BERT: A Contextual Language Model for Classical Philology](https://mlsanity.com/paper/2009.10053/) (2020)
- [CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection in 52 Languages](https://mlsanity.com/paper/1706.09031/) (2017)
- [Morphology Matters: A Multilingual Language Modeling Analysis](https://mlsanity.com/paper/2012.06262/) (2020)
- [The SIGMORPHON 2022 Shared Task on Morpheme Segmentation](https://mlsanity.com/paper/2206.07615/) (2022)
- [Evaluating Morphological Alignment of Tokenizers in 70 Languages](https://mlsanity.com/paper/2507.06378/) (2025)
- [Egalitarian Language Representation in Language Models: It All Begins with Tokenizers](https://mlsanity.com/paper/2409.11501/) (2024)
- [Sui Generis: Large Language Models for Authorship Attribution and Verification in Latin](https://mlsanity.com/paper/2410.09245/) (2024)
- [Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment](https://mlsanity.com/paper/2508.08424/) (2025)
- [What do character-level models learn about morphology? The case of dependency parsing](https://mlsanity.com/paper/1808.09180/) (2018)
- [What is the best recipe for character-level encoder-only modelling?](https://mlsanity.com/paper/2305.05461/) (2023)
- [LatinCy: Synthetic Trained Pipelines for Latin NLP](https://mlsanity.com/paper/2305.04365/) (2023)
- [Latin Treebanks in Review: An Evaluation of Morphological Tagging Across Time](https://mlsanity.com/paper/2408.06675/) (2024)

## Built on by (most cited first, 1 of 1)

- [The Limits of BPE Tokenization in Polish: Segmentation-Flexional Forms, Grammatical Anchoring, and First-Person Stability in Inflectional Language Models](https://mlsanity.com/paper/2609.17553/) (2026)

## Abstract (arXiv)

Tokenization is a critical component of language model pretraining, yet standard tokenization methods often prioritize information-theoretical goals like high compression and low fertility rather than linguistic goals like morphological alignment. In fact, they have been shown to be suboptimal for morphologically rich languages, where tokenization quality directly impacts downstream performance. In this work, we investigate morphologically-aware tokenization for Latin, a morphologically rich language that is medium-resource in terms of pretraining data, but high-resource in terms of curated lexical resources -- a distinction that is often overlooked but critical in discussions of low-resource language modeling. We find that morphologically-guided tokenization improves overall performance on four downstream tasks. Performance gains are most pronounced for out of domain texts, highlighting our models' improved generalization ability. Our findings demonstrate the utility of linguistic resources to improve language modeling for morphologically complex languages. For low-resource languages that lack large-scale pretraining data, the development and incorporation of linguistic resources can serve as a feasible alternative to improve LM performance.