mlsanityData to 14 Oct 2026

Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process

arXiv 2407.20311 ↗First posted 29 Jul 2024cs.AI cs.CL cs.LGcited by 108

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Recent advances in language models have demonstrated their capability to solve mathematical reasoning problems, achieving near-perfect accuracy on grade-school level math benchmarks like GSM8K. In this paper, we formally study how language models solve these problems. We design a series of controlled experiments to address several fundamental questions: (1) Can language models truly develop reasoning skills, or do they simply memorize templates? (2) What is the model's hidden (mental) reasoning process? (3) Do models solve math questions using skills similar to or different from humans? (4) Do models trained on GSM8K-like datasets develop reasoning skills beyond those necessary for solving GSM8K problems? (5) What mental process causes models to make reasoning mistakes? (6) How large or deep must a model be to effectively solve GSM8K-level math questions? Our study uncovers many hidden mechanisms by which language models solve mathematical questions, providing insights that extend beyond current understandings of LLMs.

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 23 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

16 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic1 May
  2. 2023TinyGSM: achieving >80% on GSM8k with small language models14 Dec

Other references · 14

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  7. 2023Textbooks Are All You Needcited by 569
  8. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  9. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  10. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  11. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  12. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
2 more
  1. 2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
  2. 2023Length Generalization in Arithmetic Transformerscited by 66

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 24; 2025: 63; 2026: 21.

108 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 912
  2. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  3. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  4. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  5. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 240
  6. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  7. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 183
  8. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  9. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  10. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  11. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 88
  12. 2025Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thoughtcited by 77
38 more
  1. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 69
  2. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 61
  3. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 59
  4. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 57
  5. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problemscited by 50
  6. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 49
  7. 2024Theoretical limitations of multi-layer Transformercited by 45
  8. 2025Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoningcited by 40
  9. 2024Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzlescited by 38
  10. 2025What Makes a Good Diffusion Planner for Decision Making?cited by 38
  11. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 33
  12. 2024GameArena: Evaluating LLM Reasoning through Live Computer Gamescited by 32
  13. 2025Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMscited by 31
  14. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 31
  15. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 31
  16. 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29
  17. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  18. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 24
  19. 2025O2-Searcher: A Searching-based Agent Model for Open-Domain Open-Ended Question Answeringcited by 23
  20. 2025The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Directioncited by 21
  21. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  22. 2025Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shufflecited by 19
  23. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 19
  24. 2024Lessons from Studying Two-Hop Latent Reasoningcited by 17
  25. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?cited by 17
  26. 2025Depth Gives a False Sense of Privacy: LLM Internal States Inversioncited by 17
  27. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  28. 2024Do LLMs Really Think Step-by-step In Implicit Reasoning?cited by 16
  29. 2024A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoningcited by 15
  30. 2025Analyzing Memorization in Large Language Models through the Lens of Model Attributioncited by 14
  31. 2025Implicit Reasoning in Transformers is Reasoning through Shortcutscited by 14
  32. 2025Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failurecited by 13
  33. 2025Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUscited by 13
  34. 2025The Serial Scaling Hypothesiscited by 13
  35. 2024MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofscited by 12
  36. 2024LLMs Faithfully and Iteratively Compute Answers During CoT: A Systematic Analysis With Multi-step Arithmeticscited by 11
  37. 2025Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Codecited by 10
  38. 2025Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoningcited by 9

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From Next-Token to Mathematics: The Learning Dynamics of Mathematical Reasoning in Language Models
  2. 2025Large Language Models and Mathematical Reasoning Failures
  3. 2024Mathfish: Evaluating Language Model Math Reasoning via Grounding in Educational Curricula
  4. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers
  5. 2023Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions
  6. 2024Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments
  7. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges
  8. 2023Three Questions Concerning the Use of Large Language Models to Facilitate Mathematics Learning