mlsanityData to 7 Oct 2026

The Geometry of Thought: How Scale Restructures Reasoning In Large Language Models

arXiv 2601.13358 ↗First posted 19 Jan 2026cs.AI cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Scale does not uniformly improve reasoning - it restructures it. Analyzing 25,000+ chain-of-thought trajectories across four domains (Law, Science, Code, Math) and two scales (8B, 70B parameters), we discover that neural scaling laws trigger domain-specific phase transitions rather than uniform capability gains. Legal reasoning undergoes Crystallization: 45% collapse in representational dimensionality (d95: 501 -> 274), 31% increase in trajectory alignment, and 10x manifold untangling. Scientific and mathematical reasoning remain Liquid - geometrically invariant despite 9x parameter increase. Code reasoning forms a discrete Lattice of strategic modes (silhouette: 0.13 -> 0.42). This geometry predicts learnability. We introduce Neural Reasoning Operators - learned mappings from initial to terminal hidden states. In crystalline legal reasoning, our operator achieves 63.6% accuracy on held-out tasks via probe decoding, predicting reasoning endpoints without traversing intermediate states. We further identify a universal oscillatory signature (coherence ~ -0.4) invariant across domains and scales, suggesting attention and feedforward layers drive reasoning through opposing dynamics. These findings establish that the cost of thought is determined not by task difficulty but by manifold geometry - offering a blueprint for inference acceleration where topology permits.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 46 of its 58 references that are in the map. First posted on arXiv 19 Jan 2026.

Builds on

46 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025REMA: A Unified Reasoning Manifold Framework for Interpreting Large Language Model26 Sep

Other references · 45

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
33 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  7. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  8. 2022In-context Learning and Induction Headscited by 930
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  10. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  11. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  13. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  14. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  15. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  16. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  17. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  18. 2021Neural Operator: Learning Maps Between Function Spacescited by 413
  19. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  20. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  21. 2022Confident Adaptive Language Modelingcited by 297
  22. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  23. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  24. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  25. 2022Predictability and Surprise in Large Generative Modelscited by 209
  26. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  27. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  28. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  29. 2023The emergence of clusters in self-attention dynamicscited by 133
  30. 2023Why think step by step? Reasoning emerges from the locality of experiencecited by 103
  31. 2019Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamicscited by 38
  32. 2025A Statistical Physics of Language Model Reasoningcited by 7
  33. 2024Geometric Analysis of Reasoning Trajectories: A Phase Space Approach to Understanding Valid and Invalid Multi-Hop Reasoning in LLMscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Emergent Manifold Separability during Reasoning in Large Language Models
  2. 2025The Geometry of Reasoning: Flowing Logics in Representation Space
  3. 2026Reasoning Models Don't Just Think Longer, They Move Differently
  4. 2026The Geometric Price of Discrete Logic: Context-driven Manifold Dynamics of Number Representations
  5. 2026Reasoning emerges from constrained inference manifolds in large language models
  6. 2026The Spectral Geometry of Thought: Phase Transitions, Instruction Reversal, Token-Level Dynamics, and Perfect Correctness Prediction in How Transformers Reason
  7. 2026Unveiling the Reasoning Process of Large Language Models
  8. 2026Beneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs