# Evaluating Morphological Alignment of Tokenizers in 70 Languages

arXiv: https://arxiv.org/abs/2507.06378 · HTML: https://mlsanity.com/paper/2507.06378/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 118 references that are in the map.
- First posted on arXiv 8 Jul 2025.

## Builds on (most cited first, 30 of 37)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://mlsanity.com/paper/2305.13707/) (2023)
- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [Tokenization Is More Than Compression](https://mlsanity.com/paper/2402.18376/) (2024)
- [MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2504.02768/) (2025)
- [Getting the most out of your tokenizer for pre-training and domain adaptation](https://mlsanity.com/paper/2402.01035/) (2024)
- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://mlsanity.com/paper/2101.00403/) (2021)
- [SuperBPE: Space Travel for Language Models](https://mlsanity.com/paper/2503.13423/) (2025)
- [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022)
- [Why do language models perform worse for morphologically complex languages?](https://mlsanity.com/paper/2411.14198/) (2024)
- [Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance](https://mlsanity.com/paper/2403.06265/) (2024)
- [Tokenization and the Noiseless Channel](https://mlsanity.com/paper/2306.16842/) (2023)
- [Greed is All You Need: An Evaluation of Tokenizer Inference Methods](https://mlsanity.com/paper/2403.01289/) (2024)
- [An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks](https://mlsanity.com/paper/2010.02534/) (2020)
- [Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier](https://mlsanity.com/paper/2504.00178/) (2025)
- [KinyaBERT: a Morphology-aware Kinyarwanda Language Model](https://mlsanity.com/paper/2203.08459/) (2022)
- [MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies](https://mlsanity.com/paper/2502.00894/) (2025)
- [Morphological and Language-Agnostic Word Segmentation for NMT](https://mlsanity.com/paper/1806.05482/) (2018)
- [The Effectiveness of Morphology-aware Segmentation in Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2103.11189/) (2021)
- [What Drives Performance in Multilingual Language Models?](https://mlsanity.com/paper/2404.19159/) (2024)

## Built on by (most cited first, 22 of 22)

- [Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization](https://mlsanity.com/paper/2508.04796/) (2025)
- [Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment](https://mlsanity.com/paper/2508.08424/) (2025)
- [Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models](https://mlsanity.com/paper/2601.13260/) (2026)
- [MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment](https://mlsanity.com/paper/2606.27019/) (2026)
- [Which Pieces Does Unigram Tokenization Really Need?](https://mlsanity.com/paper/2512.12641/) (2025)
- [Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation](https://mlsanity.com/paper/2510.09947/) (2025)
- [TokEval: A Tokenizer Evaluation Suite](https://mlsanity.com/paper/2608.18062/) (2026)
- [Contextual morphologically-guided tokenization for Latin encoder models](https://mlsanity.com/paper/2511.09709/) (2025)
- [Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs](https://mlsanity.com/paper/2603.15773/) (2026)
- [MORFES: A Benchmark for Productive Inflectional Competence in Modern Greek](https://mlsanity.com/paper/2607.28274/) (2026)
- [Opportunities and Challenges of Natural Language Processing for Low-Resource Senegalese Languages in Social Science Research](https://mlsanity.com/paper/2601.09716/) (2025)
- [Tokenization and Morphological Fidelity in Uralic NLP: A Cross-Lingual Evaluation](https://mlsanity.com/paper/2602.04241/) (2026)
- [LangMAP: A Language-Adaptive Approach to Tokenization](https://mlsanity.com/paper/2606.23566/) (2026)
- [Explicit Boundary Markers for Subword Vocabularies](https://mlsanity.com/paper/2608.08847/) (2026)
- [What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?](https://mlsanity.com/paper/2608.17325/) (2026)
- [SuTRA : Structurally-Unified Tokenization with Root Awareness](https://mlsanity.com/paper/2608.18087/) (2026)
- [MoirfEolas and Cr'iochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphology](https://mlsanity.com/paper/2609.05022/) (2026)
- [Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition](https://mlsanity.com/paper/2609.12960/) (2026)
- [The Limits of BPE Tokenization in Polish: Segmentation-Flexional Forms, Grammatical Anchoring, and First-Person Stability in Inflectional Language Models](https://mlsanity.com/paper/2609.17553/) (2026)
- [Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali](https://mlsanity.com/paper/2609.33395/) (2026)
- [Language-Specific Effects of Tokenizer Choice in Multilingual Language Models](https://mlsanity.com/paper/2610.12144/) (2026)
- [Latent Core Tokenizer: Compress, but Meaningfully](https://mlsanity.com/paper/2610.12376/) (2026)

## Abstract (arXiv)

While tokenization is a key step in language modeling, with effects on model training and performance, it remains unclear how to effectively evaluate tokenizer quality. One proposed dimension of tokenizer quality is the extent to which tokenizers preserve linguistically meaningful subwords, aligning token boundaries with morphological boundaries within a word. We expand MorphScore (Arnett & Bergen, 2025), which previously covered 22 languages, to support a total of 70 languages. The updated MorphScore offers more flexibility in evaluation and addresses some of the limitations of the original version. We then correlate our alignment scores with downstream task performance for five pre-trained languages models on seven tasks, with at least one task in each of the languages in our sample. We find that morphological alignment does not explain very much variance in model performance, suggesting that morphological alignment alone does not measure dimensions of tokenization quality relevant to model performance.