mlsanityData to 14 Oct 2026

Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models

arXiv 2608.15129 ↗First posted 15 Aug 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We systematically compare word order preferences in decoder-only language models across 192 artificial languages and typologically diverse natural languages. On artificial languages, models exhibit a left-branching preference that aligns with neither natural language universals nor human word order learning biases. On natural languages, monolingual models show no clear base word order bias at small scales, but as data grows, a preference for right-branching subject-verb-object (SVO) languages emerges while SOV falls behind despite being the most frequent order cross-linguistically. This SVO advantage extends to multilingual models and correlates with language resource level and data quality rather than word order. Thus, the same architecture exhibits opposite preferences on artificial and natural languages, establishing that word order biases observed in practice are data-driven. Since highly-resourced languages are overwhelmingly SVO, these biases risk gradually reducing word order diversity, particularly in languages that productively use multiple word orders, with the widespread adoption of LLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 62 references that are in the map. First posted on arXiv 15 Aug 2026.

Builds on

27 of 62 references.

References · 27

  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  2. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  3. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  4. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  5. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  6. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  7. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  8. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 77
  9. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  10. 2018Are All Languages Equally Hard to Language-Model?cited by 53
  11. 2019What Kind of Language Is Hard to Language-Model?cited by 50
  12. 2021Examining the Inductive Bias of Neural Language Models with Artificial Languagescited by 40
15 more
  1. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  2. 2019Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languagescited by 37
  3. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  4. 2024A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languagescited by 25
  5. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  6. 2024URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Basecited by 21
  7. 2026How LLMs Distort Our Written Languagecited by 21
  8. 2023A Cross-Linguistic Pressure for Uniform Information Density in Word Ordercited by 17
  9. 2025Multilingual Language Model Pretraining using Machine-translated Datacited by 17
  10. 2023Communication Drives the Emergence of Language Universals in Neural Agents: Evidence from the Word-order/Case-marking Trade-offcited by 12
  11. 2024Emergent Word Order Universals from Cognitively-Motivated Language Modelscited by 11
  12. 2021On the Difficulty of Translating Free-Order Case-Marking Languagescited by 8
  13. 2024NeLLCom-X: A Comprehensive Neural-Agent Framework to Simulate Language Learning and Group Communicationcited by 8
  14. 2025Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languagescited by 4
  15. 2025On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibilitycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Models
  2. 2026Language Models Generalize to Human-like Word Order Preferences
  3. 2026Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languages
  4. 2018Do RNNs learn human-like abstract word order preferences?
  5. 2024Black Big Boxes: Tracing Adjective Order Preferences in Large Language Models
  6. 2025Language Models Largely Exhibit Human-like Constituent Ordering Preferences
  7. 2025Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Models
  8. 2024ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions