# Evaluating Computational Language Models with Scaling Properties of Natural Language

arXiv: https://arxiv.org/abs/1906.09379 · HTML: https://mlsanity.com/paper/1906.09379/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2019.

## Built on by (most cited first, 22 of 22)

- [Language Model Evaluation Beyond Perplexity](https://mlsanity.com/paper/2106.00085/) (2021)
- [Evaluating Distributional Distortion in Neural Language Modeling](https://mlsanity.com/paper/2203.12788/) (2022)
- [Semantic Compression With Large Language Models](https://mlsanity.com/paper/2304.12512/) (2023)
- [Phase transition in large language models and the criticality of natural languages](https://mlsanity.com/paper/2406.05335/) (2024)
- [Probing BERT's priors with serial reproduction chains](https://mlsanity.com/paper/2202.12226/) (2022)
- [CPopQA: Ranking Cultural Concept Popularity by LLMs](https://mlsanity.com/paper/2311.07897/) (2023)
- [LLM-Generated Natural Language Meets Scaling Laws: New Explorations and Data Augmentation Methods](https://mlsanity.com/paper/2407.00322/) (2024)
- [A Natural Bias for Language Generation Models](https://mlsanity.com/paper/2212.09686/) (2022)
- [Predict the Next Word: Humans exhibit uncertainty in this task and language models _____](https://mlsanity.com/paper/2402.17527/) (2024)
- [States of LLM-generated Texts and Phase Transitions between them](https://mlsanity.com/paper/2503.06330/) (2025)
- [The Arabic AI Fingerprint: Stylometric Analysis and Detection of Large Language Models Text](https://mlsanity.com/paper/2505.23276/) (2025)
- [Correlation Dimension of Auto-Regressive Large Language Models](https://mlsanity.com/paper/2510.21258/) (2025)
- [Form and Meaning in Intrinsic Multilingual Evaluations](https://mlsanity.com/paper/2601.10580/) (2026)
- [Rethinking the Relationship between the Power Law and Hierarchical Structures](https://mlsanity.com/paper/2505.04984/) (2025)
- [SkewRoute: Training-Free LLM Routing for Knowledge Graph Retrieval-Augmented Generation via Score Skewness of Retrieved Context](https://mlsanity.com/paper/2505.23841/) (2025)
- [Strahler Number of Natural Language Sentences in Comparison with Random Trees](https://mlsanity.com/paper/2307.02697/) (2023)
- [Overfitting has a limitation: a model-independent generalization gap bound based on R'enyi entropy](https://mlsanity.com/paper/2506.00182/) (2025)
- [Scale-free Characteristics of Multilingual Legal Texts and the Limitations of LLMs](https://mlsanity.com/paper/2509.17367/) (2025)
- [A Knowledge-Based Language Model: Deducing Grammatical Knowledge in a Multi-Agent Language Acquisition Simulation](https://mlsanity.com/paper/2512.02195/) (2025)
- [Escaping Mode Collapse in LLM Generation via Geometric Regulation](https://mlsanity.com/paper/2605.00435/) (2026)
- [STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules](https://mlsanity.com/paper/2610.12599/) (2026)
- [The Geometry of Language Model Logits: Generic Structure and Learned Departures](https://mlsanity.com/paper/2610.18520/) (2026)

## Abstract (arXiv)

In this article, we evaluate computational models of natural language with respect to the universal statistical behaviors of natural language. Statistical mechanical analyses have revealed that natural language text is characterized by scaling properties, which quantify the global structure in the vocabulary population and the long memory of a text. We study whether five scaling properties (given by Zipf's law, Heaps' law, Ebeling's method, Taylor's law, and long-range correlation analysis) can serve for evaluation of computational models. Specifically, we test n-gram language models, a probabilistic context-free grammar (PCFG), language models based on Simon/Pitman-Yor processes, neural language models, and generative adversarial networks (GANs) for text generation. Our analysis reveals that language models based on recurrent neural networks (RNNs) with a gating mechanism (i.e., long short-term memory, LSTM; a gated recurrent unit, GRU; and quasi-recurrent neural networks, QRNNs) are the only computational models that can reproduce the long memory behavior of natural language. Furthermore, through comparison with recently proposed model-based evaluation methods, we find that the exponent of Taylor's law is a good indicator of model quality.