mlsanityData to 14 Oct 2026

Who Reasons in the Large Language Models?

arXiv 2505.20993 ↗First posted 27 May 2025cs.CL cs.AIcited by 5

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Despite the impressive performance of large language models (LLMs), the process of endowing them with new capabilities--such as mathematical reasoning--remains largely empirical and opaque. A critical open question is whether reasoning abilities stem from the entire model, specific modules, or are merely artifacts of overfitting. In this work, we hypothesize that the reasoning capabilities in well-trained LLMs are primarily attributed to the output projection module (oproj) in the Transformer's multi-head self-attention (MHSA) mechanism. To support this hypothesis, we introduce Stethoscope for Networks (SfN), a suite of diagnostic tools designed to probe and analyze the internal behaviors of LLMs. Using SfN, we provide both circumstantial and empirical evidence suggesting that oproj plays a central role in enabling reasoning, whereas other modules contribute more to fluent dialogue. These findings offer a new perspective on LLM interpretability and open avenues for more targeted training strategies, potentially enabling more efficient and specialized LLMs.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 35 of its 55 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

35 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  10. 2016Layer Normalizationcited by 5,669
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2024Qwen2.5 Technical Reportcited by 4,942
22 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  8. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  9. 2025s1: Simple test-time scalingcited by 1,341
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  12. 2024OpenAI o1 System Cardcited by 1,244
  13. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  14. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  15. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  16. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  17. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  18. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  19. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  20. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  21. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
  22. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillationcited by 10
  2. 2025TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummiescited by 6
  3. 2025Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learningcited by 1
  4. 2026Transcoder Adapters for Reasoning-Model Diffingcited by 1
  5. 2026Understanding Affective Adaptation in Multimodal Foundation Models: Emergent Functional Specializationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  2. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  3. 2026Understanding Large Language Models
  4. 2024Attention Heads of Large Language Models: A Survey
  5. 2025Reasoning Capabilities and Invariability of Large Language Models
  6. 2025Large Language and Reasoning Models are Shallow Disjunctive Reasoners
  7. 2022Towards Reasoning in Large Language Models: A Survey
  8. 2024LLM2: Let Large Language Models Harness System 2 Reasoning