# Do Neural Nets Learn Statistical Laws behind Natural Language?

arXiv: https://arxiv.org/abs/1707.04848 · HTML: https://mlsanity.com/paper/1707.04848/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2017.

## Built on by (most cited first, 16 of 16)

- [Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://mlsanity.com/paper/2407.13623/) (2024)
- [Language Model Evaluation Beyond Perplexity](https://mlsanity.com/paper/2106.00085/) (2021)
- [Evaluating Distributional Distortion in Neural Language Modeling](https://mlsanity.com/paper/2203.12788/) (2022)
- [Phase transition in large language models and the criticality of natural languages](https://mlsanity.com/paper/2406.05335/) (2024)
- [Probing BERT's priors with serial reproduction chains](https://mlsanity.com/paper/2202.12226/) (2022)
- [Smoothing Entailment Graphs with Language Models](https://mlsanity.com/paper/2208.00318/) (2022)
- [CPopQA: Ranking Cultural Concept Popularity by LLMs](https://mlsanity.com/paper/2311.07897/) (2023)
- [Evolution of Concepts in Language Model Pre-Training](https://mlsanity.com/paper/2509.17196/) (2025)
- [Modeling the Unigram Distribution](https://mlsanity.com/paper/2106.02289/) (2021)
- [A Natural Bias for Language Generation Models](https://mlsanity.com/paper/2212.09686/) (2022)
- [States of LLM-generated Texts and Phase Transitions between them](https://mlsanity.com/paper/2503.06330/) (2025)
- [Pragmatic Constraint on Distributional Semantics](https://mlsanity.com/paper/2211.11041/) (2022)
- [Form and Meaning in Intrinsic Multilingual Evaluations](https://mlsanity.com/paper/2601.10580/) (2026)
- [Rethinking the Relationship between the Power Law and Hierarchical Structures](https://mlsanity.com/paper/2505.04984/) (2025)
- [Overfitting has a limitation: a model-independent generalization gap bound based on R'enyi entropy](https://mlsanity.com/paper/2506.00182/) (2025)
- [Scale-Dependent Semantic Dynamics Revealed by Allan Deviation](https://mlsanity.com/paper/2601.21678/) (2026)

## Abstract (arXiv)

The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its effectiveness and of a limitation of neural networks for language engineering. Precisely, we demonstrate that a neural language model based on long short-term memory (LSTM) effectively reproduces Zipf's law and Heaps' law, two representative statistical properties underlying natural language. We discuss the quality of reproducibility and the emergence of Zipf's law and Heaps' law as training progresses. We also point out that the neural language model has a limitation in reproducing long-range correlation, another statistical property of natural language. This understanding could provide a direction for improving the architectures of neural networks.