mlsanityData to 14 Oct 2026

Vocabulary shapes cross-lingual variation of word-order learnability in language models

arXiv 2603.19427 ↗First posted 19 Mar 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Why do some languages like Czech permit free word order, while others like English do not? We address this question by pretraining transformer language models on a spectrum of synthetic word-order variants of natural languages. We observe that greater word-order irregularity consistently raises model surprisal, indicating reduced learnability. Sentence reversal, however, affects learnability only weakly. A coarse distinction of free- (e.g., Czech and Finnish) and fixed-word-order languages (e.g., English and French) does not explain cross-lingual variation. Instead, the structure of the word and subword vocabulary strongly predicts the model surprisal. Overall, vocabulary structure emerges as a key driver of computational word-order learnability across languages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 73 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

26 of 73 references.

References · 26

  1. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  2. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 73
  3. 2018Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processingcited by 63
  4. 2018Are All Languages Equally Hard to Language-Model?cited by 53
  5. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  6. 2019What Kind of Language Is Hard to Language-Model?cited by 50
  7. 2022Word Order Does Matter (And Shuffled Language Models Know It)cited by 32
  8. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  9. 2022When classifying grammatical role, BERT doesn't care about word order... except when it matterscited by 24
  10. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  11. 2025Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMscited by 20
  12. 2023A Cross-Linguistic Pressure for Uniform Information Density in Word Ordercited by 17
14 more
  1. 2019Automatically Extracting Challenge Sets for Non local Phenomena in Neural Machine Translationcited by 15
  2. 2025Information Locality as an Inductive Bias for Neural Language Modelscited by 14
  3. 2025Confounding Factors in Relating Model Performance to Morphologycited by 14
  4. 2024A Principled Framework for Evaluating on Typologically Diverse Languagescited by 12
  5. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  6. 2022Crosslinguistic word order variation reflects evolutionary pressures of dependency and information localitycited by 10
  7. 2024Multilingual Gradient Word-Order Typology from Universal Dependenciescited by 9
  8. 2021On the Difficulty of Translating Free-Order Case-Marking Languagescited by 8
  9. 2025Large Language Model probabilities cannot distinguish between possible and impossible languagecited by 5
  10. 2024Swap distance minimization beyond entropy minimization in word order variationcited by 4
  11. 2025Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languagescited by 4
  12. 2025Pico: A Modular Framework for Hypothesis-Driven Small Language Model Researchcited by 3
  13. 2022Efficient and accurate inference for mixtures of Mallows models with Spearman distancecited by 2
  14. 2025Vocabulary embeddings organize linguistic structure early in language model trainingcited by 2

Built on by

Not cited yet.