# Confounding Factors in Relating Model Performance to Morphology

arXiv: https://arxiv.org/abs/2511.01380 · HTML: https://mlsanity.com/paper/2511.01380/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 59 references that are in the map.
- First posted on arXiv 3 Nov 2025.

## Builds on (most cited first, 25 of 25)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language](https://mlsanity.com/paper/2506.20920/) (2025)
- [Tokenization Is More Than Compression](https://mlsanity.com/paper/2402.18376/) (2024)
- [Language Model Evaluation Beyond Perplexity](https://mlsanity.com/paper/2106.00085/) (2021)
- [Why do language models perform worse for morphologically complex languages?](https://mlsanity.com/paper/2411.14198/) (2024)
- [Tokenization and the Noiseless Channel](https://mlsanity.com/paper/2306.16842/) (2023)
- [Are All Languages Equally Hard to Language-Model?](https://mlsanity.com/paper/1806.03743/) (2018)
- [What Kind of Language Is Hard to Language-Model?](https://mlsanity.com/paper/1906.04726/) (2019)
- [MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling](https://mlsanity.com/paper/2403.10691/) (2024)
- [Morphology Matters: A Multilingual Language Modeling Analysis](https://mlsanity.com/paper/2012.06262/) (2020)
- [A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation](https://mlsanity.com/paper/1905.10453/) (2019)
- [PyThaiNLP: Thai Natural Language Processing in Python](https://mlsanity.com/paper/2312.04649/) (2023)
- [A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages](https://mlsanity.com/paper/2403.00686/) (2024)
- [Linguistically Motivated Vocabulary Reduction for Neural Machine Translation from Turkish to English](https://mlsanity.com/paper/1707.09879/) (2017)
- [fugashi, a Tool for Tokenizing Japanese in Python](https://mlsanity.com/paper/2010.06858/) (2020)
- [Where's the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation](https://mlsanity.com/paper/2305.18893/) (2023)
- [How Much is Enough? The Diminishing Returns of Tokenization Training Data](https://mlsanity.com/paper/2502.20273/) (2025)
- [What do complexity measures measure? Correlating and validating corpus-based measures of morphological complexity](https://mlsanity.com/paper/2204.05056/) (2022)
- [How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology?](https://mlsanity.com/paper/2109.01100/) (2021)

## Built on by (most cited first, 14 of 14)

- [Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation](https://mlsanity.com/paper/2608.25089/) (2026)
- [Form and Meaning in Intrinsic Multilingual Evaluations](https://mlsanity.com/paper/2601.10580/) (2026)
- [TokEval: A Tokenizer Evaluation Suite](https://mlsanity.com/paper/2608.18062/) (2026)
- [On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibility](https://mlsanity.com/paper/2511.08139/) (2025)
- [Assessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Models](https://mlsanity.com/paper/2602.03551/) (2026)
- [QQ: A Language Metadata Toolkit for Multilingual NLP](https://mlsanity.com/paper/2603.00620/) (2026)
- [Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP](https://mlsanity.com/paper/2605.13436/) (2026)
- [Structural priors for data-efficient language learning](https://mlsanity.com/paper/2609.11505/) (2026)
- [Dialects of Translationese Shape Language Model Learning](https://mlsanity.com/paper/2602.16469/) (2026)
- [Vocabulary shapes cross-lingual variation of word-order learnability in language models](https://mlsanity.com/paper/2603.19427/) (2026)
- [What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?](https://mlsanity.com/paper/2608.17325/) (2026)
- [Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation](https://mlsanity.com/paper/2608.30541/) (2026)
- [When Tokenization is Secretly Output Supervision](https://mlsanity.com/paper/2609.01386/) (2026)
- [MoirfEolas and Cr'iochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphology](https://mlsanity.com/paper/2609.05022/) (2026)

## Abstract (arXiv)

The extent to which individual language characteristics influence tokenization and language modeling is an open question. Differences in morphological systems have been suggested as both unimportant and crucial to consider (Cotterell et al., 2018; Gerz et al., 2018a; Park et al., 2021, inter alia). We argue this conflicting evidence is due to confounding factors in experimental setups, making it hard to compare results and draw conclusions. We identify confounding factors in analyses trying to answer the question of whether, and how, morphology relates to language modeling. Next, we re-assess three hypotheses by Arnett & Bergen (2025) for why modeling agglutinative languages results in higher perplexities than fusional languages: they look at morphological alignment of tokenization, tokenization efficiency, and dataset size. We show that each conclusion includes confounding factors. Finally, we introduce token bigram metrics as an intrinsic way to predict the difficulty of causal language modeling, and find that they are gradient proxies for morphological complexity that do not require expert annotation. Ultimately, we outline necessities to reliably answer whether, and how, morphology relates to language modeling.