mlsanityData to 14 Oct 2026

Why do language models perform worse for morphologically complex languages?

arXiv 2411.14198 ↗First posted 21 Nov 2024cs.CLcited by 62

Not on the map. Why?

Abstract (arXiv)

Language models perform differently across languages. It has been previously suggested that morphological typology may explain some of this variability (Cotterell et al., 2018). We replicate previous analyses and find additional new evidence for a performance gap between agglutinative and fusional languages, where fusional languages, such as English, tend to have better language modeling performance than morphologically more complex languages like Turkish. We then propose and test three possible causes for this performance gap: morphological alignment of tokenizers, tokenization quality, and disparities in dataset sizes and measurement. To test the morphological alignment hypothesis, we present MorphScore, a tokenizer evaluation metric, and supporting datasets for 22 languages. We find some evidence that tokenization quality explains the performance gap, but none for the role of morphological alignment. Instead we find that the performance gap is most reduced when training datasets are of equivalent size across language types, but only when scaled according to the so-called "byte-premium" -- the different encoding efficiencies of different languages and orthographies. These results suggest that no language is harder or easier for a language model to learn on the basis of its morphological typology. Differences in performance can be attributed to disparities in dataset size. These results bear on ongoing efforts to improve performance for low-performing and under-resourced languages.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 34 of its 103 references that are in the map. First posted on arXiv 21 Nov 2024.

Builds on

34 of 103 references.

References · 34

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  5. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  6. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  7. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  8. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  9. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  10. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  11. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  12. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
22 more
  1. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  2. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 66
  3. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  4. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  5. 2022Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP Worldcited by 55
  6. 2018Are All Languages Equally Hard to Language-Model?cited by 53
  7. 2018UniMorph 2.0: Universal Morphologycited by 42
  8. 2020SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflectioncited by 39
  9. 2020Morphology Matters: A Multilingual Language Modeling Analysiscited by 37
  10. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  11. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  12. 2024A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languagescited by 25
  13. 2020Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encodingcited by 22
  14. 2022Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languagescited by 20
  15. 2018Morphological and Language-Agnostic Word Segmentation for NMTcited by 16
  16. 2021The Effectiveness of Morphology-aware Segmentation in Low-Resource Neural Machine Translationcited by 16
  17. 2024Lexically Grounded Subword Segmentationcited by 13
  18. 2023CompoundPiece: Evaluating and Improving Decompounding Performance of Language Modelscited by 7
  19. 2023Explicit Morphological Knowledge Improves Pre-training of Language Models for Hebrewcited by 5
  20. 2024Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreementcited by 5
  21. 2021Morpheme Boundary Detection & Grammatical Feature Prediction for Gujarati : Dataset & Modelcited by 3
  22. 2024J-UniMorph: Japanese Morphological Annotation through the Universal Feature Schemacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 28; 2026: 33.

62 papers.

Sort:

By area:

  1. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  2. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 22
  3. 2025Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMscited by 20
  4. 2025Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizationscited by 17
  5. 2025Confounding Factors in Relating Model Performance to Morphologycited by 14
  6. 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Expertscited by 12
  7. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  8. 2025TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairscited by 9
  9. 2025Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignmentcited by 9
  10. 2025MorphTok: Morphologically Grounded Tokenization for Indian Languagescited by 7
  11. 2025Explaining and Mitigating Crosslingual Tokenizer Inequitiescited by 7
  12. 2025Subword models struggle with word learning, but surprisal hides itcited by 6
38 more
  1. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 6
  2. 2024The Impact of Token Granularity on the Predictive Power of Language Model Surprisalcited by 5
  3. 2025Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMscited by 5
  4. 2026MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignmentcited by 5
  5. 2025Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languagescited by 4
  6. 2026DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detectioncited by 4
  7. 2025UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?cited by 3
  8. 2025Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratingscited by 3
  9. 2025Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossiblecited by 3
  10. 2026Preferences for Idiomatic Language are Acquired Slowly -- and Forgotten Quickly: A Case Study on Swedishcited by 3
  11. 2026Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamicscited by 3
  12. 2026Apples to Apples? Towards Comparable Crosslingual Language Model Evaluationcited by 3
  13. 2025Limitations of Religious Data and the Importance of the Target Domain: Towards Machine Translation for Guinea-Bissau Creolecited by 2
  14. 2025Synthetic Fluency: Hallucinations, Confabulations, and the Creation of Irish Words in LLM-Generated Translationscited by 2
  15. 2025Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insightscited by 2
  16. 2025SEA-BED: How Do Embedding Models Represent Southeast Asian Languages?cited by 2
  17. 2025Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Modelscited by 2
  18. 2026Form and Meaning in Intrinsic Multilingual Evaluationscited by 2
  19. 2026Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabetcited by 2
  20. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  21. 2025Redefining Evaluation Standards: A Unified Framework for Evaluating the Korean Capabilities of Language Modelscited by 1
  22. 2025Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performancecited by 1
  23. 2025Contextual morphologically-guided tokenization for Latin encoder modelscited by 1
  24. 2026Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Featurescited by 1
  25. 2026Assessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Modelscited by 1
  26. 2026Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMscited by 1
  27. 2026Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLPcited by 1
  28. 2026MORFES: A Benchmark for Productive Inflectional Competence in Modern Greekcited by 1
  29. 2026Skill Issue: Are Skills Language-Invariant in LLMs?cited by 1
  30. 2025Towards Typologically Aware Rescoring to Mitigate Unfaithfulness in Lower-Resource Languagescited by 0
  31. 2025Construction and educational application of a linguistically grounded dependency treebank for Uyghurcited by 0
  32. 2025MoVoC: Morphology-Aware Subword Construction for Geez Script Languagescited by 0
  33. 2025Lemma Dilemma: On Lemma Generation Without Domain- or Language-Specific Training Datacited by 0
  34. 2026BYOL: Bring Your Own Language Into LLMscited by 0
  35. 2026Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosetscited by 0
  36. 2026Typologically-Informed Candidate Reranking for LLM-based Translation into Low-Resource Languagescited by 0
  37. 2026LASQ: A Low-resource Aspect-based Sentiment Quadruple Extraction Datasetcited by 0
  38. 2026Probing for Reading Timescited by 0

50 most cited of 62 All 62 (JSON)