# Mutual Information Scaling and Expressive Power of Sequence Models

arXiv: https://arxiv.org/abs/1905.04271 · HTML: https://mlsanity.com/paper/1905.04271/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2019.

## Built on by (most cited first, 14 of 14)

- [Large Sequence Models for Sequential Decision-Making: A Survey](https://mlsanity.com/paper/2306.13945/) (2023)
- [A Framework for Demonstrating Practical Quantum Advantage: Racing Quantum against Classical Generative Models](https://mlsanity.com/paper/2303.15626/) (2023)
- [Tensor networks and efficient descriptions of classical data](https://mlsanity.com/paper/2103.06872/) (2021)
- [Phase transition in large language models and the criticality of natural languages](https://mlsanity.com/paper/2406.05335/) (2024)
- [L2M: Mutual Information Scaling Law for Long-Context Language Modeling](https://mlsanity.com/paper/2503.04725/) (2025)
- [Self-attention Presents Low-dimensional Knowledge Graph Embeddings for Link Prediction](https://mlsanity.com/paper/2112.10644/) (2021)
- [Recurrent neural network wave functions for Rydberg atom arrays on kagome lattice](https://mlsanity.com/paper/2405.20384/) (2024)
- [Supplementing Recurrent Neural Network Wave Functions with Symmetry and Annealing to Improve Accuracy](https://mlsanity.com/paper/2207.14314/) (2022)
- [Investigating Topological Order using Recurrent Neural Networks](https://mlsanity.com/paper/2303.11207/) (2023)
- [States of LLM-generated Texts and Phase Transitions between them](https://mlsanity.com/paper/2503.06330/) (2025)
- [Rethinking the Relationship between the Power Law and Hierarchical Structures](https://mlsanity.com/paper/2505.04984/) (2025)
- [Adaptive Neural Quantum States: A Recurrent Neural Network Perspective](https://mlsanity.com/paper/2507.18700/) (2025)
- [Geometry-Induced Long-Range Correlations in Recurrent Neural Network Quantum States](https://mlsanity.com/paper/2604.08661/) (2026)
- [The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora](https://mlsanity.com/paper/2608.09093/) (2026)

## Abstract (arXiv)

Sequence models assign probabilities to variable-length sequences such as natural language texts. The ability of sequence models to capture temporal dependence can be characterized by the temporal scaling of correlation and mutual information. In this paper, we study the mutual information of recurrent neural networks (RNNs) including long short-term memories and self-attention networks such as Transformers. Through a combination of theoretical study of linear RNNs and empirical study of nonlinear RNNs, we find their mutual information decays exponentially in temporal distance. On the other hand, Transformers can capture long-range mutual information more efficiently, making them preferable in modeling sequences with slow power-law mutual information, such as natural languages and stock prices. We discuss the connection of these results with statistical mechanics. We also point out the non-uniformity problem in many natural language datasets. We hope this work provides a new perspective in understanding the expressive power of sequence models and shed new light on improving the architecture of them.