# Emergent Word Order Universals from Cognitively-Motivated Language Models

arXiv: https://arxiv.org/abs/2402.12363 · HTML: https://mlsanity.com/paper/2402.12363/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 94 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Neural Networks and the Chomsky Hierarchy](https://mlsanity.com/paper/2207.02098/) (2022)
- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [On the Practical Computational Power of Finite Precision RNNs for Language Recognition](https://mlsanity.com/paper/1805.04908/) (2018)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Recurrent Neural Network Grammars](https://mlsanity.com/paper/1602.07776/) (2016)
- [Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2301.11796/) (2023)
- [Anti-efficient encoding in emergent communication](https://mlsanity.com/paper/1905.12561/) (2019)
- [RNNs can generate bounded hierarchical languages with optimal memory](https://mlsanity.com/paper/2010.07515/) (2020)
- [Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations](https://mlsanity.com/paper/1603.04351/) (2016)
- [Are All Languages Equally Hard to Language-Model?](https://mlsanity.com/paper/1806.03743/) (2018)
- [What Kind of Language Is Hard to Language-Model?](https://mlsanity.com/paper/1906.04726/) (2019)
- [Finding Syntax in Human Encephalography with Beam Search](https://mlsanity.com/paper/1806.04127/) (2018)
- [Examining the Inductive Bias of Neural Language Models with Artificial Languages](https://mlsanity.com/paper/2106.01044/) (2021)
- [Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times](https://mlsanity.com/paper/2402.02255/) (2024)
- [Context Limitations Make Neural Language Models More Human-Like](https://mlsanity.com/paper/2205.11463/) (2022)
- [Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages](https://mlsanity.com/paper/1911.03329/) (2019)
- [Emergent Communication: Generalization and Overfitting in Lewis Games](https://mlsanity.com/paper/2209.15342/) (2022)
- [Effective Inference for Generative Neural Parsing](https://mlsanity.com/paper/1707.08976/) (2017)
- [Structural Supervision Improves Learning of Non-Local Grammatical Dependencies](https://mlsanity.com/paper/1903.00943/) (2019)
- [Emergent Communication for Understanding Human Language Evolution: What's Missing?](https://mlsanity.com/paper/2204.10590/) (2022)
- [What they do when in doubt: a study of inductive biases in seq2seq learners](https://mlsanity.com/paper/2006.14953/) (2020)
- [The Galactic Dependencies Treebanks: Getting More Data by Synthesizing New Languages](https://mlsanity.com/paper/1710.03838/) (2017)
- [Word-order biases in deep-agent emergent communication](https://mlsanity.com/paper/1905.12330/) (2019)
- [What Do Recurrent Neural Network Grammars Learn About Syntax?](https://mlsanity.com/paper/1611.05774/) (2016)
- [Effective Batching for Recurrent Neural Network Grammars](https://mlsanity.com/paper/2105.14822/) (2021)

## Built on by (most cited first, 11 of 11)

- [Can Language Models Learn Typologically Implausible Languages?](https://mlsanity.com/paper/2502.12317/) (2025)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [NeLLCom-X: A Comprehensive Neural-Agent Framework to Simulate Language Learning and Group Communication](https://mlsanity.com/paper/2407.13999/) (2024)
- [Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages](https://mlsanity.com/paper/2510.12722/) (2025)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [Assessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Models](https://mlsanity.com/paper/2602.03551/) (2026)
- [Differences in Typological Alignment in Language Models' Treatment of Differential Argument Marking](https://mlsanity.com/paper/2602.17653/) (2026)
- [What Kind of Language is Easy to Language-Model Under Curriculum Learning?](https://mlsanity.com/paper/2604.26844/) (2026)
- [ConlangBench: Exploring Language Knowledge and Learning in LLMs through Diverse Constructed Languages](https://mlsanity.com/paper/2608.03505/) (2026)
- [Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models](https://mlsanity.com/paper/2608.15129/) (2026)
- [Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languages](https://mlsanity.com/paper/2610.02040/) (2026)

## Abstract (arXiv)

The world's languages exhibit certain so-called typological or implicational universals; for example, Subject-Object-Verb (SOV) languages typically use postpositions. Explaining the source of such biases is a key goal of linguistics. We study word-order universals through a computational simulation with language models (LMs). Our experiments show that typologically-typical word orders tend to have lower perplexity estimated by LMs with cognitively plausible biases: syntactic biases, specific parsing strategies, and memory limitations. This suggests that the interplay of cognitive biases and predictability (perplexity) can explain many aspects of word-order universals. It also showcases the advantage of cognitively-motivated LMs, typically employed in cognitive modeling, in the simulation of language universals.