# Phase Transitions in Large Language Models and the O(N) Model

arXiv: https://arxiv.org/abs/2501.16241 · HTML: https://mlsanity.com/paper/2501.16241/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2025.

## Built on by (most cited first, 12 of 12)

- [Detailed balance in large language model-driven agents](https://mlsanity.com/paper/2512.10047/) (2025)
- [Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models](https://mlsanity.com/paper/2501.18280/) (2025)
- [Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review](https://mlsanity.com/paper/2504.18346/) (2025)
- [HessFormer: Hessians at Foundation Scale](https://mlsanity.com/paper/2505.11564/) (2025)
- [Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation](https://mlsanity.com/paper/2503.22547/) (2025)
- [Towards Worst-Case Guarantees with Scale-Aware Interpretability](https://mlsanity.com/paper/2602.05184/) (2026)
- [LLMs Meet Finance: Fine-Tuning Foundation Models for the Open FinLLM Leaderboard](https://mlsanity.com/paper/2504.13125/) (2025)
- [Generative Criticality in Large Language Model Temperature Scaling](https://mlsanity.com/paper/2606.06238/) (2026)
- [Temperature-driven inversion and nonlinear dynamics in ChatGPT-like AIs](https://mlsanity.com/paper/2608.00939/) (2026)
- [Lecture Notes on Statistical Physics and Neural Networks](https://mlsanity.com/paper/2605.06394/) (2026)
- [Many-body Tipping Dynamics of ChatGPT-like AIs](https://mlsanity.com/paper/2607.25279/) (2026)
- [Pre-registered tests of solid-state-physics-inspired LLM compression: a cluster-level negative result at small-language-model scale](https://mlsanity.com/paper/2609.34292/) (2026)

## Abstract (arXiv)

Large language models (LLMs) exhibit unprecedentedly rich scaling behaviors. In physics, scaling behavior is closely related to phase transitions, critical phenomena, and field theory. To investigate the phase transition phenomena in LLMs, we reformulated the Transformer architecture as an O(N) model. Our study reveals two distinct phase transitions corresponding to the temperature used in text generation and the model's parameter size, respectively. The first phase transition enables us to estimate the internal dimension of the model, while the second phase transition is of textithigher-depth and signals the emergence of new capabilities. As an application, the energy of the O(N) model can be used to evaluate whether an LLM's parameters are sufficient to learn the training data.