# A Cross-Linguistic Pressure for Uniform Information Density in Word Order

arXiv: https://arxiv.org/abs/2306.03734 · HTML: https://mlsanity.com/paper/2306.03734/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 80 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Revisiting the Uniform Information Density Hypothesis](https://mlsanity.com/paper/2109.11635/) (2021)
- [Examining the Inductive Bias of Neural Language Models with Artificial Languages](https://mlsanity.com/paper/2106.01044/) (2021)
- [Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages](https://mlsanity.com/paper/1903.06400/) (2019)
- [A Measure-Theoretic Characterization of Tight Language Models](https://mlsanity.com/paper/2212.10502/) (2022)
- [A surprisal--duration trade-off across and within the world's languages](https://mlsanity.com/paper/2109.15000/) (2021)
- [Please Mind the Root: Decoding Arborescences for Dependency Parsing](https://mlsanity.com/paper/2010.02550/) (2020)
- [How (Non-)Optimal is the Lexicon?](https://mlsanity.com/paper/2104.14279/) (2021)
- [Crosslinguistic word order variation reflects evolutionary pressures of dependency and information locality](https://mlsanity.com/paper/2206.04239/) (2022)
- [Disambiguatory Signals are Stronger in Word-initial Positions](https://mlsanity.com/paper/2102.02183/) (2021)
- [Are crossing dependencies really scarce?](https://mlsanity.com/paper/1703.08324/) (2017)
- [Human languages order information efficiently](https://mlsanity.com/paper/1510.02823/) (2015)
- [Grammatical cues to subjecthood are redundant in a majority of simple clauses across languages](https://mlsanity.com/paper/2201.12911/) (2022)

## Built on by (most cited first, 17 of 17)

- [Can Language Models Learn Typologically Implausible Languages?](https://mlsanity.com/paper/2502.12317/) (2025)
- [Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs](https://mlsanity.com/paper/2502.18795/) (2025)
- [Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse](https://mlsanity.com/paper/2410.16062/) (2024)
- [UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactions](https://mlsanity.com/paper/2504.20304/) (2025)
- [Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible](https://mlsanity.com/paper/2510.07178/) (2025)
- [Is Information Density Uniform when Utterances are Grounded on Perception and Discourse?](https://mlsanity.com/paper/2602.14653/) (2026)
- [Reverse-Engineering the Reader](https://mlsanity.com/paper/2410.13086/) (2024)
- [Strategic resource allocation in memory encoding: An efficiency principle shaping language processing](https://mlsanity.com/paper/2503.14728/) (2025)
- [Is Structure Dependence Shaped for Efficient Communication?: A Case Study on Coordination](https://mlsanity.com/paper/2410.10556/) (2024)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [That's Optional: A Contemporary Exploration of "that" Omission in English Subordinate Clauses](https://mlsanity.com/paper/2405.20833/) (2024)
- [CSSL: Contrastive Self-Supervised Learning for Dependency Parsing on Relatively Free Word Ordered and Morphologically Rich Low Resource Languages](https://mlsanity.com/paper/2410.06944/) (2024)
- [Vocabulary shapes cross-lingual variation of word-order learnability in language models](https://mlsanity.com/paper/2603.19427/) (2026)
- [Expect the Unexpected? Testing the Surprisal of Salient Entities](https://mlsanity.com/paper/2604.10724/) (2026)
- [Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models](https://mlsanity.com/paper/2608.15129/) (2026)
- [Factors Influencing the Emergence of Dependency Length Minimization in Neural Agent Simulations](https://mlsanity.com/paper/2609.06025/) (2026)
- [An Interpretable Linguistically-Grounded Analysis of Scene Saliency in Movie Screenplays](https://mlsanity.com/paper/2610.15544/) (2026)

## Abstract (arXiv)

While natural languages differ widely in both canonical word order and word order flexibility, their word orders still follow shared cross-linguistic statistical patterns, often attributed to functional pressures. In the effort to identify these pressures, prior work has compared real and counterfactual word orders. Yet one functional pressure has been overlooked in such investigations: the uniform information density (UID) hypothesis, which holds that information should be spread evenly throughout an utterance. Here, we ask whether a pressure for UID may have influenced word order patterns cross-linguistically. To this end, we use computational models to test whether real orders lead to greater information uniformity than counterfactual orders. In our empirical study of 10 typologically diverse languages, we find that: (i) among SVO languages, real word orders consistently have greater uniformity than reverse word orders, and (ii) only linguistically implausible counterfactual orders consistently exceed the uniformity of real orders. These findings are compatible with a pressure for information uniformity in the development and usage of natural languages.