Is Mathematical Problem-Solving Expertise in Large Language Models Associated with Assessment Performance?
Not on the map. Why?
Abstract (arXiv)
Large Language Models (LLMs) are increasingly used in math education not only as problem solvers but also as assessors of learners' reasoning. However, it remains unclear whether stronger math problem-solving ability is associated with stronger step-level assessment performance. This study examines that relationship using the GSM8K and MATH subsets of PROCESSBENCH, a human-annotated benchmark for identifying the earliest erroneous step in mathematical reasoning. We evaluate two LLM-based math tutor agent settings, instantiated with GPT-4 and GPT-5, in two independent tasks on the same math problems: solving the original problem and assessing a benchmark-provided solution by predicting the earliest erroneous step. Results show a consistent within-model pattern: assessment accuracy is substantially higher on math problem items the same model solved correctly than on items it solved incorrectly, with statistically significant associations across both models and datasets. At the same time, assessment remains more difficult than direct problem solving, especially on error-present solutions. These findings suggest that math problem-solving expertise supports stronger assessment performance, but reliable step-level diagnosis also requires additional capabilities such as step tracking, monitoring, and precise error localization. The results have implications for the design and evaluation of AI-supported Adaptive Instructional Systems (AISs) for formative assessment in math education.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 21 references that are in the map. First posted on arXiv 26 Mar 2026.
Builds on
12 of 21 references.
References · 12
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2023Let's Verify Step by Stepcited by 3,598
- 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
- 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
- 2024Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutorscited by 39
- 2025Beyond Final Answers: Evaluating Large Language Models for Math Tutoringcited by 23
- 2024Evaluating GPT-4 at Grading Handwritten Solutions in Math Examscited by 10
- 2024SPL: A Socratic Playground for Learning Powered by Large Language Modelcited by 9
- 2024Autograding Mathematical Induction Proofs with Natural Language Processingcited by 4
- 2025Mathematical Computation and Reasoning Errors by Large Language Modelscited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Three Questions Concerning the Use of Large Language Models to Facilitate Mathematics Learning
- 2025Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles
- 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data
- 2025Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work
- 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process
- 2024From Next-Token to Mathematics: The Learning Dynamics of Mathematical Reasoning in Language Models
- 2024Benchmarking Large Language Models for Math Reasoning Tasks
- 2023Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions