# Are All Languages Equally Hard to Language-Model?

arXiv: https://arxiv.org/abs/1806.03743 · HTML: https://mlsanity.com/paper/1806.03743/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2018.

## Built on by (most cited first, 30 of 53)

- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [Why do language models perform worse for morphologically complex languages?](https://mlsanity.com/paper/2411.14198/) (2024)
- [Examining the Inductive Bias of Neural Language Models with Artificial Languages](https://mlsanity.com/paper/2106.01044/) (2021)
- [MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling](https://mlsanity.com/paper/2403.10691/) (2024)
- [Dissociating language and thought in large language models](https://mlsanity.com/paper/2301.06627/) (2023)
- [Zero-shot cross-lingual transfer language selection using linguistic similarity](https://mlsanity.com/paper/2301.13720/) (2023)
- [Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs](https://mlsanity.com/paper/2502.18795/) (2025)
- [Evaluating Morphological Compositional Generalization in Large Language Models](https://mlsanity.com/paper/2410.12656/) (2024)
- [gzip Predicts Data-dependent Scaling Laws](https://mlsanity.com/paper/2405.16684/) (2024)
- [Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5](https://mlsanity.com/paper/2410.11627/) (2024)
- [Confounding Factors in Relating Model Performance to Morphology](https://mlsanity.com/paper/2511.01380/) (2025)
- [Estimating the Entropy of Linguistic Distributions](https://mlsanity.com/paper/2204.01469/) (2022)
- [A Cognitive Regularizer for Language Modeling](https://mlsanity.com/paper/2105.07144/) (2021)
- [DivEMT: Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languages](https://mlsanity.com/paper/2205.12215/) (2022)
- [Emergent Word Order Universals from Cognitively-Motivated Language Models](https://mlsanity.com/paper/2402.12363/) (2024)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution](https://mlsanity.com/paper/2310.14840/) (2023)
- [MorphPiece : A Linguistic Tokenizer for Large Language Models](https://mlsanity.com/paper/2307.07262/) (2023)
- [Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment](https://mlsanity.com/paper/2508.08424/) (2025)
- [On the Difficulty of Translating Free-Order Case-Marking Languages](https://mlsanity.com/paper/2107.06055/) (2021)
- [Transfer Language Selection for Zero-Shot Cross-Lingual Abusive Language Detection](https://mlsanity.com/paper/2206.00962/) (2022)
- [GPT-SW3: An Autoregressive Language Model for the Nordic Languages](https://mlsanity.com/paper/2305.12987/) (2023)
- [Xihe: Scalable Zero-Shot Time Series Learner Via Hierarchical Interleaved Block Attention](https://mlsanity.com/paper/2510.21795/) (2025)
- [Modeling the Unigram Distribution](https://mlsanity.com/paper/2106.02289/) (2021)
- [The interplay between language similarity and script on a novel multi-layer Algerian dialect corpus](https://mlsanity.com/paper/2105.07400/) (2021)
- [Using AntiPatterns to avoid MLOps Mistakes](https://mlsanity.com/paper/2107.00079/) (2021)
- [Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages](https://mlsanity.com/paper/2510.12722/) (2025)

## Abstract (arXiv)

For general modeling methods applied to diverse languages, a natural question is: how well should we expect our models to work on languages with differing typological profiles? In this work, we develop an evaluation framework for fair cross-linguistic comparison of language models, using translated text so that all models are asked to predict approximately the same information. We then conduct a study on 21 languages, demonstrating that in some languages, the textual expression of the information is harder to predict with both n-gram and LSTM language models. We show complex inflectional morphology to be a cause of performance differences among languages.