mlsanityData to 14 Oct 2026

The Geometry of Reasoning: Flowing Logics in Representation Space

arXiv 2510.09782 ↗First posted 10 Oct 2025cs.AI cs.CL cs.LGcited by 21

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We study how large language models (LLMs) ``think'' through their representation space. We propose a novel geometric framework that models an LLM's reasoning as flows -- embedding trajectories evolving where logic goes. We disentangle logical structure from semantics by employing the same natural deduction propositions with varied semantic carriers, allowing us to test whether LLMs internalize logic beyond surface form. This perspective connects reasoning with geometric quantities such as position, velocity, and curvature, enabling formal analysis in representation and concept spaces. Our theory establishes: (1) LLM reasoning corresponds to smooth flows in representation space, and (2) logical statements act as local controllers of these flows' velocities. Using learned representation proxies, we design controlled experiments to visualize and quantify reasoning flows, providing empirical validation of our theoretical framework. Our findings indicate that training solely via next-token prediction can lead LLMs to internalize logical invariants as higher-order geometry in representation space, challenging the ``stochastic parrot'' argument. Experiments across Qwen and LLaMA model families further suggest the presence of a general, possibly universal, representational law underlying machine understanding and human linguistic regularities, largely independent of specific training recipes or model architectures. Our work serves as both a conceptual foundation and practical tools for studying reasoning phenomena, offering a new lens for interpretability and formal analysis of LLMs' behavior.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 66 of its 87 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

66 of 87 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?22 May
  2. 2025Qwen3 Technical Report14 May
  3. 2025Can Large Language Models Learn Formal Logic? A Data-Driven Training and Evaluation Framework28 Apr
  4. 2025Logical Reasoning in Large Language Models: A Survey13 Feb
  5. 2025An Analysis for Reasoning Bias of Language Models with Small Initialization5 Feb
  6. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus19 Nov
  7. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models23 Apr
  8. 2024Understanding Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation5 Feb
  9. 2024Rethinking Interpretability in the Era of Large Language Models30 Jan

Other references · 57

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  4. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  5. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  6. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  7. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  9. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  10. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  11. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  12. 2023Language Models Represent Space and Timecited by 316
38 more
  1. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  2. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  3. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  4. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 183
  5. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  6. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 159
  7. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 156
  8. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 129
  9. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  10. 2025Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Spacecited by 117
  11. 2023The geometry of hidden representations of large transformer modelscited by 112
  12. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 108
  13. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 104
  14. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 94
  15. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  16. 2024On the Origins of Linear Representations in Large Language Modelscited by 79
  17. 2025Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thoughtcited by 77
  18. 2025Efficient Reasoning with Hidden Thinkingcited by 75
  19. 2025Understanding Reasoning in Thinking Language Models via Steering Vectorscited by 72
  20. 2025Language Models Use Trigonometry to Do Additioncited by 70
  21. 2025SEAL: Steerable Reasoning Calibration of Large Language Models for Freecited by 68
  22. 2022A Logic for Expressing Log-Precision Transformerscited by 67
  23. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  24. 2023Concept Algebra for (Score-Based) Text-Controlled Generative Modelscited by 59
  25. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 58
  26. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 57
  27. 2024Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thoughtcited by 56
  28. 2025Harnessing the Universal Geometry of Embeddingscited by 55
  29. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  30. 2025The Origins of Representation Manifolds in Large Language Modelscited by 51
  31. 2024Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformerscited by 39
  32. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 37
  33. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 35
  34. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 35
  35. 2025Mitigating Overthinking in Large Reasoning Models via Manifold Steeringcited by 33
  36. 2025Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Propertiescited by 29
  37. 2024When Text Embedding Meets Large Language Model: A Comprehensive Surveycited by 19
  38. 2024Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoningcited by 18

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 20.

21 papers.

Sort:

By area:

  1. 2026LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signalscited by 24
  2. 2026Coding Agents are Effective Long-Context Processorscited by 12
  3. 2026Are Your Reasoning Models Reasoning or Guessing? A Mechanistic Analysis of Hierarchical Reasoning Modelscited by 10
  4. 2026Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoningcited by 10
  5. 2026Beyond Scalars: Evaluating and Understanding LLM Reasoning via Geometric Progress and Stabilitycited by 5
  6. 2026Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information in Latent Tokenscited by 4
  7. 2026Limited Reasoning Space: The cage of long-horizon reasoning in LLMscited by 3
  8. 2026Think Before You Lie: How Reasoning Leads to Honestycited by 2
  9. 2026SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topologycited by 2
  10. 2026Reasoning Models Don't Just Think Longer, They Move Differentlycited by 2
  11. 2026How Do Large Language Models Learn Concepts During Continual Pre-Training?cited by 1
  12. 2026Directional Reasoning Trajectory Change (DRTC): Identifying Critical Trace Segments in Reasoning Modelscited by 1
9 more
  1. 2026The Geometric Price of Discrete Logic: Context-driven Manifold Dynamics of Number Representationscited by 1
  2. 2026Reasoning Models Know What's Important, and Encode It in Their Activationscited by 1
  3. 2026Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Samplingcited by 1
  4. 2026An Analysis of Residual-Stream Geometry Across Transformer Depthcited by 1
  5. 2026When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Modelscited by 1
  6. 2025Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Modelscited by 0
  7. 2026Reasoning Errors Have a Region and a Direction in the Residual-Stream Trajectory of LLMscited by 0
  8. 2026Logical subspace in LLMscited by 0
  9. 2026The Geometry of Contextual Relations: Language Models Address Facts by Order of Mentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs
  2. 2026Reasoning emerges from constrained inference manifolds in large language models
  3. 2026Unveiling the Reasoning Process of Large Language Models
  4. 2024Reasoning in Large Language Models: A Geometric Perspective
  5. 2026The Geometry of Thought: How Scale Restructures Reasoning In Large Language Models
  6. 2024Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large Language Models
  7. 2026Geometric Latent Reasoning Induces Shorter Generations in LLMs
  8. 2026Emergent Manifold Separability during Reasoning in Large Language Models