mlsanityData to 7 Oct 2026

Reasoning in Large Language Models: A Geometric Perspective

arXiv 2407.02678 ↗First posted 2 Jul 2024cs.AI cs.CLcited by 4

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The advancement of large language models (LLMs) for real-world applications hinges critically on enhancing their reasoning capabilities. In this work, we explore the reasoning abilities of large language models (LLMs) through their geometrical understanding. We establish a connection between the expressive power of LLMs and the density of their self-attention graphs. Our analysis demonstrates that the density of these graphs defines the intrinsic dimension of the inputs to the MLP blocks. We demonstrate through theoretical analysis and toy examples that a higher intrinsic dimension implies a greater expressive capacity of the LLM. We further provide empirical evidence linking this geometric framework to recent advancements in methods aimed at enhancing the reasoning capabilities of LLMs.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 20 of its 41 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

20 of 41 references.

References · 20

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  5. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  6. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  7. 2021Scaling Laws for Transfercited by 265
  8. 2024Many-Shot In-Context Learningcited by 203
  9. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 186
  10. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  11. 2023Teaching Arithmetic to Small Transformerscited by 121
  12. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
8 more
  1. 2024Transformers, parallel computation, and logarithmic depthcited by 75
  2. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 64
  3. 2023Mimetic Initialization of Self-Attention Layerscited by 52
  4. 2023Transformers learn through gradual rank increasecited by 52
  5. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 39
  6. 2022Systematic Generalization and Emergent Structures in Transformers Trained on Structured Taskscited by 19
  7. 2023Uncovering hidden geometry in Transformers via disentangling position and contextcited by 15
  8. 2023Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 3.

4 papers.

  1. 2024Understanding Reasoning in Chain-of-Thought from the Hopfieldian Viewcited by 10
  2. 2026Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden Statescited by 7
  3. 2026The Linear Centroids Hypothesis: Features as Directions Learned by Local Expertscited by 3
  4. 2026Runtime Burden Allocation for Structured LLM Routing in Agentic Expert Systems: A Full-Factorial Cross-Backend Methodologycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large Language Models
  2. 2024Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Path
  3. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  4. 2025A Survey of Scaling in Large Language Model Reasoning
  5. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  6. 2026Are Large Language Models Suitable for Graph Computation? Progress and Prospects
  7. 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
  8. 2023GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approach