mlsanityData to 14 Oct 2026

Evaluating Morphological Alignment of Tokenizers in 70 Languages

arXiv 2507.06378 ↗First posted 8 Jul 2025cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

While tokenization is a key step in language modeling, with effects on model training and performance, it remains unclear how to effectively evaluate tokenizer quality. One proposed dimension of tokenizer quality is the extent to which tokenizers preserve linguistically meaningful subwords, aligning token boundaries with morphological boundaries within a word. We expand MorphScore (Arnett & Bergen, 2025), which previously covered 22 languages, to support a total of 70 languages. The updated MorphScore offers more flexibility in evaluation and addresses some of the limitations of the original version. We then correlate our alignment scores with downstream task performance for five pre-trained languages models on seven tasks, with at least one task in each of the languages in our sample. We find that morphological alignment does not explain very much variance in model performance, suggesting that morphological alignment alone does not measure dimensions of tokenization quality relevant to model performance.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 37 of its 118 references that are in the map. First posted on arXiv 8 Jul 2025.

Builds on

37 of 118 references.

References · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  8. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  9. 2023Language Modeling Is Compressioncited by 225
  10. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  11. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  12. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
25 more
  1. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  2. 2024Tokenization Is More Than Compressioncited by 89
  3. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  4. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 66
  5. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  6. 2025SuperBPE: Space Travel for Language Modelscited by 63
  7. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  8. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  9. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  10. 2023Tokenization and the Noiseless Channelcited by 59
  11. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  12. 2020An Empirical Study of Tokenization Strategies for Various Korean NLP Taskscited by 33
  13. 2025Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barriercited by 32
  14. 2022KinyaBERT: a Morphology-aware Kinyarwanda Language Modelcited by 27
  15. 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 25
  16. 2018Morphological and Language-Agnostic Word Segmentation for NMTcited by 16
  17. 2021The Effectiveness of Morphology-aware Segmentation in Low-Resource Neural Machine Translationcited by 16
  18. 2024What Drives Performance in Multilingual Language Models?cited by 15
  19. 2024Two Counterexamples to Tokenization and the Noiseless Channelcited by 13
  20. 2024Lexically Grounded Subword Segmentationcited by 13
  21. 2024Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?cited by 13
  22. 2023MorphPiece : A Linguistic Tokenizer for Large Language Modelscited by 9
  23. 2024Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challengecited by 8
  24. 2025Splintering Nonconcatenative Languages for Better Tokenizationcited by 6
  25. 2024Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreementcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 16.

22 papers.

Sort:

  1. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 20
  2. 2025Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignmentcited by 9
  3. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 6
  4. 2026MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignmentcited by 5
  5. 2025Which Pieces Does Unigram Tokenization Really Need?cited by 4
  6. 2025Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluationcited by 3
  7. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  8. 2025Contextual morphologically-guided tokenization for Latin encoder modelscited by 1
  9. 2026Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMscited by 1
  10. 2026MORFES: A Benchmark for Productive Inflectional Competence in Modern Greekcited by 1
  11. 2025Opportunities and Challenges of Natural Language Processing for Low-Resource Senegalese Languages in Social Science Researchcited by 0
  12. 2026Tokenization and Morphological Fidelity in Uralic NLP: A Cross-Lingual Evaluationcited by 0
10 more
  1. 2026LangMAP: A Language-Adaptive Approach to Tokenizationcited by 0
  2. 2026Explicit Boundary Markers for Subword Vocabulariescited by 0
  3. 2026What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?cited by 0
  4. 2026SuTRA : Structurally-Unified Tokenization with Root Awarenesscited by 0
  5. 2026MoirfEolas and Cr'iochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphologycited by 0
  6. 2026Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Propositioncited by 0
  7. 2026The Limits of BPE Tokenization in Polish: Segmentation-Flexional Forms, Grammatical Anchoring, and First-Person Stability in Inflectional Language Modelscited by 0
  8. 2026Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepalicited by 0
  9. 2026Language-Specific Effects of Tokenizer Choice in Multilingual Language Modelscited by 0
  10. 2026Latent Core Tokenizer: Compress, but Meaningfullycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
  2. 2025How Language Directions Align with Token Geometry in Multilingual LLMs
  3. 2025Beyond Literal Token Overlap: Token Alignability for Multilinguality
  4. 2026How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them
  5. 2025IMPACT: Inflectional Morphology Probes Across Complex Typologies
  6. 2025The Token Tax: Systematic Bias in Multilingual Tokenization
  7. 2024Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective
  8. 2023On the Calibration of Large Language Models and Alignment