# Beyond Final Answers: Evaluating Large Language Models for Math Tutoring

arXiv: https://arxiv.org/abs/2503.16460 · HTML: https://mlsanity.com/paper/2503.16460/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 26 references that are in the map.
- First posted on arXiv 23 Feb 2025.

## Builds on (most cited first, 7 of 7)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [Evaluating Mathematical Reasoning Beyond Accuracy](https://mlsanity.com/paper/2404.05692/) (2024)
- [MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data](https://mlsanity.com/paper/2406.18321/) (2024)
- [Intelligent Tutors for Adult Learners: An Analysis of Needs and Challenges](https://mlsanity.com/paper/2412.04477/) (2024)

## Built on by (most cited first, 23 of 23)

- [From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning](https://mlsanity.com/paper/2505.15607/) (2025)
- [TutorBench: A Benchmark To Assess Tutoring Capabilities Of Large Language Models](https://mlsanity.com/paper/2510.02663/) (2025)
- [TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students](https://mlsanity.com/paper/2505.01563/) (2025)
- [Reliable Proof Generation with LLMs via Analogical Retrieval and Symbolic Verification: A Case Study in Euclidean Geometry](https://mlsanity.com/paper/2505.14479/) (2025)
- [Evaluation of LLMs for mathematical problem solving](https://mlsanity.com/paper/2506.00309/) (2025)
- [Evaluating Answer Leakage Robustness of LLM Tutors against Adversarial Student Attacks](https://mlsanity.com/paper/2604.18660/) (2026)
- [Mathematical Computation and Reasoning Errors by Large Language Models](https://mlsanity.com/paper/2508.09932/) (2025)
- [Personalized Auto-Grading and Feedback System for Constructive Geometry Tasks Using Large Language Models on an Online Math Platform](https://mlsanity.com/paper/2509.25529/) (2025)
- [MMTutorBench: The First Multimodal Benchmark for AI Math Tutoring](https://mlsanity.com/paper/2510.23477/) (2025)
- [Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most](https://mlsanity.com/paper/2605.16207/) (2026)
- [GuideAI: A Real-time Personalized Learning Solution with Adaptive Interventions](https://mlsanity.com/paper/2601.20402/) (2026)
- [Guidelines for Designing AI Technologies to Support Adult Learning](https://mlsanity.com/paper/2605.04616/) (2026)
- [The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness](https://mlsanity.com/paper/2605.05648/) (2026)
- [Learning to Prompt: Improving Student Engagement with Adaptive LLM-based High-School Tutoring](https://mlsanity.com/paper/2606.20138/) (2026)
- [What Drives Interactive Improvement from Feedback?](https://mlsanity.com/paper/2606.30774/) (2026)
- [Designing and Evaluating Chain-of-Hints for Scientific Question Answering](https://mlsanity.com/paper/2510.21087/) (2025)
- [Developing an LLM-Based Feedback System Grounded in Evidence-Centered Design to Support Physics Problem Solving](https://mlsanity.com/paper/2512.10785/) (2025)
- [Enhancing Mathematical Problem Solving in LLMs through Execution-Driven Reasoning Augmentation](https://mlsanity.com/paper/2602.03950/) (2026)
- [Is Mathematical Problem-Solving Expertise in Large Language Models Associated with Assessment Performance?](https://mlsanity.com/paper/2603.25633/) (2026)
- [Math Education Digital Shadows for Investigating Learning with GenAI: Mathematics Performance, Anxiety, and Confidence in LLMs](https://mlsanity.com/paper/2604.27618/) (2026)
- [Can Vision Language Models Be Adaptive in Mathematics Education? A Learner Model-based Rubric Study](https://mlsanity.com/paper/2605.16011/) (2026)
- [T2MM: An LLM Supported Architecture For Inquiry-Based Modeling](https://mlsanity.com/paper/2606.11210/) (2026)
- [PIVOT: Preference-based Intervention Vectors for Pedagogical Tutor Steering](https://mlsanity.com/paper/2608.07509/) (2026)

## Abstract (arXiv)

Researchers have made notable progress in applying Large Language Models (LLMs) to solve math problems, as demonstrated through efforts like GSM8k, ProofNet, AlphaGeometry, and MathOdyssey. This progress has sparked interest in their potential use for tutoring students in mathematics. However, the reliability of LLMs in tutoring contexts -- where correctness and instructional quality are crucial -- remains underexplored. Moreover, LLM problem-solving capabilities may not necessarily translate into effective tutoring support for students. In this work, we present two novel approaches to evaluate the correctness and quality of LLMs in math tutoring contexts. The first approach uses an intelligent tutoring system for college algebra as a testbed to assess LLM problem-solving capabilities. We generate benchmark problems using the tutor, prompt a diverse set of LLMs to solve them, and compare the solutions to those generated by the tutor. The second approach evaluates LLM as tutors rather than problem solvers. We employ human evaluators, who act as students seeking tutoring support from each LLM. We then assess the quality and correctness of the support provided by the LLMs via a qualitative coding process. We applied these methods to evaluate several ChatGPT models, including 3.5 Turbo, 4, 4o, o1-mini, and o1-preview. Our findings show that when used as problem solvers, LLMs generate correct final answers for 85.5% of the college algebra problems tested. When employed interactively as tutors, 90% of LLM dialogues show high-quality instructional support; however, many contain errors -- only 56.6% are entirely correct. We conclude that, despite their potential, LLMs are not yet suitable as intelligent tutors for math without human oversight or additional mechanisms to ensure correctness and quality.