mlsanityData to 7 Oct 2026

Large Language Models for Mathematical Reasoning: Progresses and Challenges

arXiv 2402.00157 ↗First posted 31 Jan 2024cs.CLcited by 283

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Mathematical reasoning serves as a cornerstone for assessing the fundamental cognitive capabilities of human intelligence. In recent times, there has been a notable surge in the development of Large Language Models (LLMs) geared towards the automated resolution of mathematical problems. However, the landscape of mathematical problem types is vast and varied, with LLM-oriented techniques undergoing evaluation across diverse datasets and settings. This diversity makes it challenging to discern the true advancements and obstacles within this burgeoning field. This survey endeavors to address four pivotal dimensions: i) a comprehensive exploration of the various mathematical problems and their corresponding datasets that have been investigated; ii) an examination of the spectrum of LLM-oriented techniques that have been proposed for mathematical problem-solving; iii) an overview of factors and concerns affecting LLMs in solving math; and iv) an elucidation of the persisting challenges within this domain. To the best of our knowledge, this survey stands as one of the first extensive examinations of the landscape of LLMs in the realm of mathematics, providing a holistic perspective on the current state, accomplishments, and future challenges in this rapidly evolving field.

Key facts. Cited by 283 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 67 of its 94 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

67 of 94 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Learning From Mistakes Makes LLM Better Reasoner31 Oct
  2. 2023Fill in the Blank: Exploring and Enhancing LLM Capabilities for Backward Reasoning in Math Word Problems3 Oct
  3. 2023LPML: LLM-Prompting Markup Language for Mathematical Reasoning21 Sep
  4. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  5. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification15 Aug
  6. 2023Reasoning in Large Language Models Through Symbolic Math Word Problems3 Aug
  7. 2023MinT: Boosting Generalization in Mathematical Reasoning via Multi-View Fine-Tuning16 Jul
  8. 2023Math Word Problem Solving by Generating Linguistic Variants of Problem Statements24 Jun
  9. 2023Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning4 Jun
  10. 2023How well do Large Language Models perform in Arithmetic tasks?16 Mar
  11. 2022A Survey of Deep Learning for Mathematical Reasoning20 Dec
  12. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Models28 Oct
3 more
  1. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models21 Oct
  2. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning29 Sep
  3. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun

Other references · 52

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
38 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  7. 2022Galactica: A Large Language Model for Sciencecited by 736
  8. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  9. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  10. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  11. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  12. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
  13. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  14. 2016Solving General Arithmetic Word Problemscited by 353
  15. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  16. 2023Mathematical Capabilities of ChatGPTcited by 263
  17. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  18. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  19. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137
  20. 2021Proof Artifact Co-training for Theorem Proving with Language Modelscited by 130
  21. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  22. 2019Learning to Prove Theorems via Interacting with Proof Assistantscited by 107
  23. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
  24. 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 89
  25. 2023GPT Can Solve Mathematical Problems Without a Calculatorcited by 68
  26. 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agentscited by 67
  27. 2020Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problemscited by 60
  28. 2023Improving Large Language Model Fine-tuning for Solving Math Problemscited by 48
  29. 2021Neural-Symbolic Solver for Math Word Problems with Auxiliary Taskscited by 38
  30. 2023An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)cited by 31
  31. 2023MathAttack: Attacking Large Language Models Towards Math Solving Abilitycited by 31
  32. 2016Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problemscited by 28
  33. 2021Math Word Problem Generation with Mathematical Consistency and Problem Context Constraintscited by 27
  34. 2023Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generationcited by 27
  35. 2019HOList: An Environment for Machine Learning of Higher-Order Theorem Provingcited by 19
  36. 2023Interpretable Math Word Problem Solution Generation Via Step-by-step Planningcited by 17
  37. 2023Three Questions Concerning the Use of Large Language Models to Facilitate Mathematics Learningcited by 9
  38. 2023Solving Math Word Problem with Problem Type Classificationcited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 70; 2025: 163; 2026: 50.

283 papers.

Sort:

By area:

  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  2. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  3. 2024Discrete Flow Matchingcited by 301
  4. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  5. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  6. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  7. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  8. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecyclecited by 129
  9. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  10. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95
  11. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  12. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
38 more
  1. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 83
  2. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  3. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  4. 2025Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipelinecited by 79
  5. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  6. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 65
  7. 2025SEAL: Steerable Reasoning Calibration of Large Language Models for Freecited by 65
  8. 2025Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?cited by 58
  9. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  10. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
  11. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  12. 2025FLEX: Continuous Agent Evolution via Forward Learning from Experiencecited by 51
  13. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 51
  14. 2025Astra: A Multi-Agent System for GPU Kernel Performance Optimizationcited by 46
  15. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  16. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 39
  17. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  18. 2024Agent-Oriented Planning in Multi-Agent Systemscited by 34
  19. 2025MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reportscited by 34
  20. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  21. 2025Mitigating Overthinking in Large Reasoning Models via Manifold Steeringcited by 32
  22. 2025NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agentscited by 32
  23. 2025MPMA: Preference Manipulation Attack Against Model Context Protocolcited by 31
  24. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  25. 2025Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Modelscited by 30
  26. 2024Easy Problems That LLMs Get Wrongcited by 29
  27. 2024Not All LLM Reasoners Are Created Equalcited by 29
  28. 2024TS-Reasoner: Domain-Oriented Time Series Inference Agents for Reasoning and Automated Analysiscited by 28
  29. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
  30. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 27
  31. 2025Adversarial Reasoning at Jailbreaking Timecited by 27
  32. 2025Large Language Models as End-to-end Combinatorial Optimization Solverscited by 27
  33. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpeningcited by 27
  34. 2025ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoningcited by 26
  35. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMscited by 25
  36. 2025ReCode: Reinforcing Code Generation with Reasoning-Process Rewardscited by 25
  37. 2025PDE-Controller: LLMs for Autoformalization and Reasoning of PDEscited by 23
  38. 2025Time-R1: Towards Comprehensive Temporal Reasoning in LLMscited by 23

50 most cited of 283 All 283 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  2. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges
  3. 2025Large Language Models and Mathematical Reasoning Failures
  4. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  5. 2025Thinking Machines: Mathematical Reasoning in the Age of LLMs
  6. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges
  7. 2025SMART: Self-Generating and Self-Validating Multi-Dimensional Assessment for LLMs' Mathematical Problem Solving
  8. 2024LLM Reasoning Engine: Specialized Training for Enhanced Mathematical Reasoning