# Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction

arXiv: https://arxiv.org/abs/2406.00755 · HTML: https://mlsanity.com/paper/2406.00755/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 36 of its 67 references that are in the map.
- First posted on arXiv 2 Jun 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 30 of 36)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Llemma: An Open Language Model For Mathematics](https://mlsanity.com/paper/2310.10631/) (2023)
- [Faithful Chain-of-Thought Reasoning](https://mlsanity.com/paper/2301.13379/) (2023)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters](https://mlsanity.com/paper/2212.10001/) (2022)
- [Inverse Scaling: When Bigger Isn't Better](https://mlsanity.com/paper/2306.09479/) (2023)
- [REFINER: Reasoning Feedback on Intermediate Representations](https://mlsanity.com/paper/2304.01904/) (2023)
- [Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification](https://mlsanity.com/paper/2308.07921/) (2023)
- [Can Large Language Models Really Improve by Self-critiquing Their Own Plans?](https://mlsanity.com/paper/2310.08118/) (2023)
- [Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance](https://mlsanity.com/paper/2305.17306/) (2023)
- [Learning From Mistakes Makes LLM Better Reasoner](https://mlsanity.com/paper/2310.20689/) (2023)
- [GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems](https://mlsanity.com/paper/2310.12397/) (2023)
- [Evaluating Language Models for Mathematics through Interactions](https://mlsanity.com/paper/2306.01694/) (2023)
- [Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning](https://mlsanity.com/paper/2306.02408/) (2023)
- [Position: AI Evaluation Should Learn from How We Test Humans](https://mlsanity.com/paper/2306.10512/) (2023)
- [An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)](https://mlsanity.com/paper/2302.13814/) (2023)
- [Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions](https://mlsanity.com/paper/2401.09395/) (2024)

## Built on by (most cited first, 30 of 45)

- [PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models](https://mlsanity.com/paper/2501.03124/) (2025)
- [A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges](https://mlsanity.com/paper/2412.11936/) (2024)
- [Number Cookbook: Number Understanding of Language Models and How to Improve It](https://mlsanity.com/paper/2411.03766/) (2024)
- [ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection](https://mlsanity.com/paper/2410.04509/) (2024)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents](https://mlsanity.com/paper/2412.00821/) (2024)
- [Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges](https://mlsanity.com/paper/2502.08680/) (2025)
- [MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detection](https://mlsanity.com/paper/2503.18132/) (2025)
- [LEMMA: Learning from Errors for MatheMatical Advancement in LLMs](https://mlsanity.com/paper/2503.17439/) (2025)
- [Solving Inequality Proofs with Large Language Models](https://mlsanity.com/paper/2506.07927/) (2025)
- [AI-Driven Virtual Teacher for Enhanced Educational Efficiency: Leveraging Large Pretrain Models for Autonomous Error Analysis and Correction](https://mlsanity.com/paper/2409.09403/) (2024)
- [Can Vision-Language Models Evaluate Handwritten Math?](https://mlsanity.com/paper/2501.07244/) (2025)
- [Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive Framework](https://mlsanity.com/paper/2501.15581/) (2025)
- [Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach](https://mlsanity.com/paper/2503.10573/) (2025)
- [Improving Human Verification of LLM Reasoning through Interactive Explanation Interfaces](https://mlsanity.com/paper/2510.22922/) (2025)
- [RMA: an Agentic System for Research-Level Mathematical Problems](https://mlsanity.com/paper/2605.22875/) (2026)
- [When Do LLMs Admit Their Mistakes? Understanding The Role Of Model Belief In Retraction](https://mlsanity.com/paper/2505.16170/) (2025)
- [ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Models](https://mlsanity.com/paper/2601.15812/) (2026)
- [Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains](https://mlsanity.com/paper/2510.18176/) (2025)
- [To Err Is Human: Systematic Quantification of Errors in Published AI Papers via LLM Analysis](https://mlsanity.com/paper/2512.05925/) (2025)
- [Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutions](https://mlsanity.com/paper/2412.16838/) (2024)
- [Mathematical Computation and Reasoning Errors by Large Language Models](https://mlsanity.com/paper/2508.09932/) (2025)
- [PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning](https://mlsanity.com/paper/2510.03185/) (2025)
- [Insight Agents: An LLM-Based Multi-Agent System for Data Insights](https://mlsanity.com/paper/2601.20048/) (2026)
- [Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code](https://mlsanity.com/paper/2507.07498/) (2025)
- [Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability](https://mlsanity.com/paper/2508.04017/) (2025)
- [Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during Multi-Hop Analysis](https://mlsanity.com/paper/2508.04699/) (2025)
- [Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigidity](https://mlsanity.com/paper/2604.22597/) (2026)
- [Small Language Models are Equation Reasoners](https://mlsanity.com/paper/2409.12393/) (2024)
- [TuringQ: Benchmarking AI Comprehension in Theory of Computation](https://mlsanity.com/paper/2410.06547/) (2024)

## Abstract (arXiv)

The rapid advancement of Large Language Models (LLMs) in the realm of mathematical reasoning necessitates comprehensive evaluations to gauge progress and inspire future directions. Existing assessments predominantly focus on problem-solving from the examinee perspective, overlooking a dual perspective of examiner regarding error identification and correction. From the examiner perspective, we define four evaluation tasks for error identification and correction along with a new dataset with annotated error types and steps. We also design diverse prompts to thoroughly evaluate eleven representative LLMs. Our principal findings indicate that GPT-4 outperforms all models, while open-source model LLaMA-2-7B demonstrates comparable abilities to closed-source models GPT-3.5 and Gemini Pro. Notably, calculation error proves the most challenging error type. Moreover, prompting LLMs with the error types can improve the average correction accuracy by 47.9%. These results reveal potential directions for developing the mathematical reasoning abilities of LLMs. Our code and dataset is available on https://github.com/LittleCirc1e/EIC.