mlsanityData to 14 Oct 2026

Emergent Word Order Universals from Cognitively-Motivated Language Models

arXiv 2402.12363 ↗First posted 19 Feb 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

The world's languages exhibit certain so-called typological or implicational universals; for example, Subject-Object-Verb (SOV) languages typically use postpositions. Explaining the source of such biases is a key goal of linguistics. We study word-order universals through a computational simulation with language models (LMs). Our experiments show that typologically-typical word orders tend to have lower perplexity estimated by LMs with cognitively plausible biases: syntactic biases, specific parsing strategies, and memory limitations. This suggests that the interplay of cognitive biases and predictability (perplexity) can explain many aspects of word-order universals. It also showcases the advantage of cognitively-motivated LMs, typically employed in cognitive modeling, in the simulation of language universals.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 36 of its 94 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

36 of 94 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  6. 2022Neural Networks and the Chomsky Hierarchycited by 217
  7. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 119
  8. 2018On the Practical Computational Power of Finite Precision RNNs for Language Recognitioncited by 117
  9. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
  10. 2016Recurrent Neural Network Grammarscited by 109
  11. 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
  12. 2019Anti-efficient encoding in emergent communicationcited by 59
24 more
  1. 2020RNNs can generate bounded hierarchical languages with optimal memorycited by 59
  2. 2016Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representationscited by 55
  3. 2018Are All Languages Equally Hard to Language-Model?cited by 53
  4. 2019What Kind of Language Is Hard to Language-Model?cited by 50
  5. 2018Finding Syntax in Human Encephalography with Beam Searchcited by 41
  6. 2021Examining the Inductive Bias of Neural Language Models with Artificial Languagescited by 40
  7. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 39
  8. 2022Context Limitations Make Neural Language Models More Human-Likecited by 34
  9. 2019Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languagescited by 31
  10. 2022Emergent Communication: Generalization and Overfitting in Lewis Gamescited by 28
  11. 2017Effective Inference for Generative Neural Parsingcited by 25
  12. 2019Structural Supervision Improves Learning of Non-Local Grammatical Dependenciescited by 24
  13. 2022Emergent Communication for Understanding Human Language Evolution: What's Missing?cited by 23
  14. 2020What they do when in doubt: a study of inductive biases in seq2seq learnerscited by 20
  15. 2017The Galactic Dependencies Treebanks: Getting More Data by Synthesizing New Languagescited by 19
  16. 2019Word-order biases in deep-agent emergent communicationcited by 18
  17. 2016What Do Recurrent Neural Network Grammars Learn About Syntax?cited by 15
  18. 2021Effective Batching for Recurrent Neural Network Grammarscited by 12
  19. 2023Communication Drives the Emergence of Language Universals in Neural Agents: Evidence from the Word-order/Case-marking Trade-offcited by 12
  20. 2018Do RNNs learn human-like abstract word order preferences?cited by 10
  21. 2021The Effect of Efficient Messaging and Input Variability on Neural-Agent Iterated Language Learningcited by 6
  22. 2021Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammarscited by 5
  23. 2020On the Branching Bias of Syntax Extracted from Pre-trained Language Modelscited by 3
  24. 2022Discourse Context Predictability Effects in Hindi Word Ordercited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 8.

11 papers.

  1. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  2. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  3. 2024NeLLCom-X: A Comprehensive Neural-Agent Framework to Simulate Language Learning and Group Communicationcited by 8
  4. 2025Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languagescited by 4
  5. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  6. 2026Assessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Modelscited by 1
  7. 2026Differences in Typological Alignment in Language Models' Treatment of Differential Argument Markingcited by 1
  8. 2026What Kind of Language is Easy to Language-Model Under Curriculum Learning?cited by 0
  9. 2026ConlangBench: Exploring Language Knowledge and Learning in LLMs through Diverse Constructed Languagescited by 0
  10. 2026Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Modelscited by 0
  11. 2026Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languagescited by 0