# Linguistic Structure from a Bottleneck on Sequential Information Processing

arXiv: https://arxiv.org/abs/2405.12109 · HTML: https://mlsanity.com/paper/2405.12109/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 158 references that are in the map.
- First posted on arXiv 20 May 2024.

## Builds on (most cited first, 11 of 11)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Multi-Agent Cooperation and the Emergence of (Natural) Language](https://mlsanity.com/paper/1612.07182/) (2016)
- [Mission: Impossible Language Models](https://mlsanity.com/paper/2401.06416/) (2024)
- [From partners to populations: A hierarchical Bayesian account of coordination and convention](https://mlsanity.com/paper/2104.05857/) (2021)
- [Predictive information in a sensory population](https://mlsanity.com/paper/1307.0225/) (2013)
- [Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication](https://mlsanity.com/paper/2111.06464/) (2021)
- [A Cross-Genre Ensemble Approach to Robust Reddit Part of Speech Tagging](https://mlsanity.com/paper/2004.14312/) (2020)
- [Memory limitations are hidden in grammar](https://mlsanity.com/paper/1908.06629/) (2019)
- [Basic syntax from speech: Spontaneous concatenation in unsupervised deep neural networks](https://mlsanity.com/paper/2305.01626/) (2023)

## Built on by (most cited first, 5 of 5)

- [Information Locality as an Inductive Bias for Neural Language Models](https://mlsanity.com/paper/2506.05136/) (2025)
- [Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible](https://mlsanity.com/paper/2510.07178/) (2025)
- [From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis](https://mlsanity.com/paper/2512.13491/) (2025)
- [Exploiting individual differences to bootstrap communication](https://mlsanity.com/paper/2504.05211/) (2025)
- [On the Dynamics of Observation and Semantics](https://mlsanity.com/paper/2602.18494/) (2026)

## Abstract (arXiv)

Human language has a distinct systematic structure, where utterances break into individually meaningful words which are combined to form phrases. We show that natural-language-like systematicity arises in codes that are constrained by a statistical measure of complexity called predictive information, also known as excess entropy. Predictive information is the mutual information between the past and future of a stochastic process. In simulations, we find that such codes break messages into groups of approximately independent features which are expressed systematically and locally, corresponding to words and phrases. Next, drawing on crosslinguistic text corpora, we find that actual human languages are structured in a way that reduces predictive information compared to baselines at the levels of phonology, morphology, syntax, and lexical semantics. Our results establish a link between the statistical and algebraic structure of language and reinforce the idea that these structures are shaped by communication under general cognitive constraints.