# Phase transition in large language models and the criticality of natural languages

arXiv: https://arxiv.org/abs/2406.05335 · HTML: https://mlsanity.com/paper/2406.05335/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 50 references that are in the map.
- First posted on arXiv 8 Jun 2024.

## Builds on (most cited first, 10 of 10)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Criticality in Formal Languages and Statistical Physics](https://mlsanity.com/paper/1606.06737/) (2016)
- [Evaluating Computational Language Models with Scaling Properties of Natural Language](https://mlsanity.com/paper/1906.09379/) (2019)
- [Release of Pre-Trained Models for the Japanese Language](https://mlsanity.com/paper/2404.01657/) (2024)
- [Do Neural Nets Learn Statistical Laws behind Natural Language?](https://mlsanity.com/paper/1707.04848/) (2017)
- [Mutual Information Scaling and Expressive Power of Sequence Models](https://mlsanity.com/paper/1905.04271/) (2019)
- [Autocorrelations Decay in Texts and Applicability Limits of Language Models](https://mlsanity.com/paper/2305.06615/) (2023)
- [Phase Transitions in the Output Distribution of Large Language Models](https://mlsanity.com/paper/2405.17088/) (2024)

## Built on by (most cited first, 13 of 13)

- [Large Language Models and Emergence: A Complex Systems Perspective](https://mlsanity.com/paper/2506.11135/) (2025)
- [Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignment](https://mlsanity.com/paper/2508.20015/) (2025)
- [States of LLM-generated Texts and Phase Transitions between them](https://mlsanity.com/paper/2503.06330/) (2025)
- [Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting](https://mlsanity.com/paper/2509.06770/) (2025)
- [Unsupervised detection of semantic correlations in big data](https://mlsanity.com/paper/2411.02126/) (2024)
- [The Disparate Impacts of Speculative Decoding](https://mlsanity.com/paper/2510.02128/) (2025)
- [Generative Criticality in Large Language Model Temperature Scaling](https://mlsanity.com/paper/2606.06238/) (2026)
- [World-Model Collapse as a Phase Transition](https://mlsanity.com/paper/2606.31399/) (2026)
- [Temperature-driven inversion and nonlinear dynamics in ChatGPT-like AIs](https://mlsanity.com/paper/2608.00939/) (2026)
- [Turbulence-like 5/3 spectral scaling in contextual representations of language as a complex system](https://mlsanity.com/paper/2604.05536/) (2026)
- [A Limit Theory of Foundation Models: A Mathematical Approach to Understanding Emergent Intelligence and Scaling Laws](https://mlsanity.com/paper/2604.24037/) (2026)
- [Escaping Mode Collapse in LLM Generation via Geometric Regulation](https://mlsanity.com/paper/2605.00435/) (2026)
- [Many-body Tipping Dynamics of ChatGPT-like AIs](https://mlsanity.com/paper/2607.25279/) (2026)

## Abstract (arXiv)

Generation of text and speech in natural languages can be modeled as a stochastic process. This idea dates back to the seminal work of Markov and, later, to that of Shannon and also underlies the recent development of large language models (LLMs). The stochastic processes corresponding to natural languages should be distinct from those that generate nonlinguistic sequences. One of the features that discriminate linguistic and nonlinguistic sequences is power-law behavior, which is universally observed across different languages. In statistical physics, such behavior suggests that natural languages are critical: They lie near a phase transition point in a parametrized space of stochastic processes. However, testing this conjecture is not straightforward. A phase transition, even if it exists, cannot be directly observed in real-world natural languages because they do not have any controllable parameters. Here, we use LLMs as controllable effective models of natural languages. Through statistical analyses of texts generated by LLMs, we find that, when a parameter analogous to physical temperature is varied, LLMs undergo a phase transition. The transition separates a low-temperature phase with complex repetitive structures in generated texts from a high-temperature phase in which LLMs generate incomprehensible texts. At the critical point between these phases, generated texts display the power-law behavior similar to that of natural languages and most closely resemble natural languages as measured by a standard metric in natural language processing. These findings strongly suggest that natural languages are indeed critical.