mlsanityData to 14 Oct 2026

A Statistical Physics of Language Model Reasoning

arXiv 2506.04374 ↗First posted 4 Jun 2025cs.AI cs.CLcited by 7

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Transformer LMs show emergent reasoning that resists mechanistic understanding. We offer a statistical physics framework for continuous-time chain-of-thought reasoning dynamics. We model sentence-level hidden state trajectories as a stochastic dynamical system on a lower-dimensional manifold. This drift-diffusion system uses latent regime switching to capture diverse reasoning phases, including misaligned states or failures. Empirical trajectories (8 models, 7 benchmarks) show a rank-40 projection (balancing variance capture and feasibility) explains ~50% variance. We find four latent reasoning regimes. An SLDS model is formulated and validated to capture these features. The framework enables low-cost reasoning simulation, offering tools to study and predict critical transitions like misaligned states or other LM failures.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 22 of its 42 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

22 of 42 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023Mistral 7Bcited by 3,164
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
10 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  2. 2022In-context Learning and Induction Headscited by 964
  3. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  4. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  5. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  6. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  7. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  8. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 109
  9. 2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
  10. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  2. 2025The wall confronting large language modelscited by 8
  3. 2026Is my model "mind blurting"? Interpreting the dynamics of reasoning tokens with Recurrence Quantification Analysis (RQA)cited by 1
  4. 2026Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Samplingcited by 1
  5. 2026The Geometry of Thought: How Scale Restructures Reasoning In Large Language Modelscited by 0
  6. 2026Phantom Transitions in Language Model Fine-Tuning: A Density-Matrix Analysiscited by 0
  7. 2026The Depth Flow of Token Representations Is Nonlinear and Does Not Descend Its Own Densitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dissecting Failure Dynamics in Large Language Model Reasoning
  2. 2026LLM Reasoning Is Latent, Not the Chain of Thought
  3. 2026Mean-Field Dynamics of Chain-of-Thought Reasoning in Large Language Models
  4. 2026Spatiotemporal Hidden-State Dynamics as a Signature of Internal Reasoning in Large Language Models
  5. 2025A Survey on Latent Reasoning
  6. 2026Emergent Search and Backtracking in Latent Reasoning Models
  7. 2026Answer-Distribution Trajectories: A Stochastic-Dynamics View of LLM Reasoning
  8. 2026Reasoning emerges from constrained inference manifolds in large language models