mlsanityData to 7 Oct 2026

Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction

arXiv 2406.00755 ↗First posted 2 Jun 2024cs.CL cs.AI cs.LGcited by 45

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

The rapid advancement of Large Language Models (LLMs) in the realm of mathematical reasoning necessitates comprehensive evaluations to gauge progress and inspire future directions. Existing assessments predominantly focus on problem-solving from the examinee perspective, overlooking a dual perspective of examiner regarding error identification and correction. From the examiner perspective, we define four evaluation tasks for error identification and correction along with a new dataset with annotated error types and steps. We also design diverse prompts to thoroughly evaluate eleven representative LLMs. Our principal findings indicate that GPT-4 outperforms all models, while open-source model LLaMA-2-7B demonstrates comparable abilities to closed-source models GPT-3.5 and Gemini Pro. Notably, calculation error proves the most challenging error type. Moreover, prompting LLMs with the error types can improve the average correction accuracy by 47.9%. These results reveal potential directions for developing the mathematical reasoning abilities of LLMs. Our code and dataset is available on https://github.com/LittleCirc1e/EIC.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 67 references that are in the map. First posted on arXiv 2 Jun 2024.

Builds on

36 of 67 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions17 Jan
  2. 2023VerityMath: Advancing Mathematical Reasoning by Self-Verification Through Unit Consistency13 Nov
  3. 2023Learning From Mistakes Makes LLM Better Reasoner31 Oct
  4. 2023Llemma: An Open Language Model For Mathematics16 Oct
  5. 2023Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions3 Oct
  6. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification15 Aug
  7. 2023Reasoning in Large Language Models Through Symbolic Math Word Problems3 Aug
  8. 2023Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning4 Jun
  9. 2022Language Models are Multilingual Chain-of-Thought Reasoners6 Oct

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2023Mistral 7Bcited by 3,065
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  10. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  12. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
15 more
  1. 2023Faithful Chain-of-Thought Reasoningcited by 285
  2. 2023Mathematical Capabilities of ChatGPTcited by 263
  3. 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repaircited by 225
  4. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  5. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  6. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  7. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 107
  8. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  9. 2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problemscited by 88
  10. 2023Evaluating Language Models for Mathematics through Interactionscited by 69
  11. 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
  12. 2023An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)cited by 31
  13. 2023STREET: A Multi-Task Structured Reasoning and Explanation Benchmarkcited by 13
  14. 2021SYNFIX: Automatically Fixing Syntax Errors using Compiler Diagnosticscited by 8
  15. 2018Don't Panic! Better, Fewer, Syntax Errors for LR Parserscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 26; 2026: 11.

45 papers.

Sort:

By area:

  1. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 98
  2. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  3. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 41
  4. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  5. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  6. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  7. 2025MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detectioncited by 24
  8. 2024Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agentscited by 22
  9. 2025Solving Inequality Proofs with Large Language Modelscited by 21
  10. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMscited by 20
  11. 2024AI-Driven Virtual Teacher for Enhanced Educational Efficiency: Leveraging Large Pretrain Models for Autonomous Error Analysis and Correctioncited by 12
  12. 2025Can Vision-Language Models Evaluate Handwritten Math?cited by 10
33 more
  1. 2025Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive Frameworkcited by 10
  2. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approachcited by 8
  3. 2025Improving Human Verification of LLM Reasoning through Interactive Explanation Interfacescited by 7
  4. 2025When Do LLMs Admit Their Mistakes? Understanding The Role Of Model Belief In Retractioncited by 6
  5. 2026ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Modelscited by 6
  6. 2026RMA: an Agentic System for Research-Level Mathematical Problemscited by 6
  7. 2025Local Coherence or Global Validity? Investigating RLVR Traces in Math Domainscited by 5
  8. 2025To Err Is Human: Systematic Quantification of Errors in Published AI Papers via LLM Analysiscited by 5
  9. 2024Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutionscited by 4
  10. 2025Mathematical Computation and Reasoning Errors by Large Language Modelscited by 4
  11. 2025PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoningcited by 4
  12. 2026Insight Agents: An LLM-Based Multi-Agent System for Data Insightscited by 4
  13. 2025Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Codecited by 3
  14. 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Abilitycited by 3
  15. 2025Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during Multi-Hop Analysiscited by 3
  16. 2026Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigiditycited by 3
  17. 2024Small Language Models are Equation Reasonerscited by 2
  18. 2024TuringQ: Benchmarking AI Comprehension in Theory of Computationcited by 2
  19. 2025Mis-prompt: Benchmarking Large Language Models for Proactive Error Handlingcited by 2
  20. 2025Judge Before Answer: Can MLLM Discern the False Premise in Question?cited by 2
  21. 2026Semantic Needles in Document Haystacks: Sensitivity Testing of LLM-as-a-Judge Similarity Scoringcited by 2
  22. 2025Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematicscited by 1
  23. 2025MMRefine: Unveiling the Obstacles to Robust Refinement in Multimodal Large Language Modelscited by 1
  24. 2026LLMs for Game Theory: Entropy-Guided In-Context Learning and Adaptive CoT Reasoningcited by 1
  25. 2026What Drives Interactive Improvement from Feedback?cited by 1
  26. 2026HPFA: Hypergraph-Based Paired Failure Attribution for LLM Reasoningcited by 1
  27. 2025PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoningcited by 0
  28. 2025Parrot: A Training Pipeline Enhances Both Program CoT and Natural Language CoT for Reasoningcited by 0
  29. 2025Is GPT-OSS All You Need? Benchmarking Large Language Models for Financial Intelligence and the Surprising Efficiency Paradoxcited by 0
  30. 2025Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profilescited by 0
  31. 2026When Tokenization is Secretly Output Supervisioncited by 0
  32. 2026Math Reasoning in LLMs is Organized by Approach, Not Topiccited by 0
  33. 2026Large Language Models are Approximate Survival Estimatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Mathematical Reasoning Capabilities in Large Language Models via Reasoning Process Error Classification
  2. 2024Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning
  3. 2025Can LLMs textitunderstand Math? -- Exploring the Pitfalls in Mathematical Reasoning
  4. 2025Large Language Models and Mathematical Reasoning Failures
  5. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  6. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  7. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
  8. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges