# MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors

arXiv: https://arxiv.org/abs/2502.18940 · HTML: https://mlsanity.com/paper/2502.18940/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems](https://mlsanity.com/paper/2305.14536/) (2023)
- [The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues](https://mlsanity.com/paper/2205.07540/) (2022)
- [Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors](https://mlsanity.com/paper/2412.09416/) (2024)
- [Dialog Inpainting: Turning Documents into Dialogs](https://mlsanity.com/paper/2205.09073/) (2022)
- [Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes](https://mlsanity.com/paper/2310.10648/) (2023)
- [CLASS: A Design Framework for building Intelligent Tutoring Systems based on Learning Science principles](https://mlsanity.com/paper/2305.13272/) (2023)
- [Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions](https://mlsanity.com/paper/2106.03873/) (2021)
- [The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts](https://mlsanity.com/paper/2211.11772/) (2022)
- [Automatic Generation of Socratic Subquestions for Teaching Math Word Problems](https://mlsanity.com/paper/2211.12835/) (2022)
- [Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors](https://mlsanity.com/paper/2407.09136/) (2024)
- [Opportunities and Challenges in Neural Dialog Tutoring](https://mlsanity.com/paper/2301.09919/) (2023)
- [The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues](https://mlsanity.com/paper/2306.06941/) (2023)
- [Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System](https://mlsanity.com/paper/2404.17460/) (2024)
- [MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions](https://mlsanity.com/paper/2405.19444/) (2024)
- [Book2Dial: Generating Teacher-Student Interactions from Textbooks for Cost-Effective Development of Educational Chatbots](https://mlsanity.com/paper/2403.03307/) (2024)
- [Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging](https://mlsanity.com/paper/2406.11709/) (2024)
- [The Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Education](https://mlsanity.com/paper/2404.02444/) (2024)

## Built on by (most cited first, 30 of 49)

- [LLM Agents for Education: Advances and Applications](https://mlsanity.com/paper/2503.11733/) (2025)
- [PIIvot: A Lightweight NLP Anonymization Framework for Question-Anchored Tutoring Dialogues](https://mlsanity.com/paper/2505.16931/) (2025)
- [Benchmarking the Pedagogical Knowledge of Large Language Models](https://mlsanity.com/paper/2506.18710/) (2025)
- [Pedagogy-driven Evaluation of Generative AI-powered Intelligent Tutoring Systems](https://mlsanity.com/paper/2510.22581/) (2025)
- [DualSchool: How Reliable are LLMs for Optimization Education?](https://mlsanity.com/paper/2505.21775/) (2025)
- [FeedEval: Pedagogically Aligned Evaluation of LLM-Generated Essay Feedback](https://mlsanity.com/paper/2601.04574/) (2026)
- [EduEVAL-DB: A Role-Based Dataset for Pedagogical Risk Evaluation in Educational Explanations](https://mlsanity.com/paper/2602.15531/) (2026)
- [Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Education](https://mlsanity.com/paper/2601.14560/) (2026)
- [ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluation](https://mlsanity.com/paper/2507.12674/) (2025)
- [Shaping Explanations: Semantic Reward Modeling with Encoder-Only Transformers for GRPO](https://mlsanity.com/paper/2509.13081/) (2025)
- [MMTutorBench: The First Multimodal Benchmark for AI Math Tutoring](https://mlsanity.com/paper/2510.23477/) (2025)
- [How well do Large Language Models Recognize Instructional Moves? Establishing Baselines for Foundation Models in Educational Discourse](https://mlsanity.com/paper/2512.19903/) (2025)
- [From Solver to Tutor: Evaluating the Pedagogical Intelligence of LLMs with KMP-Bench](https://mlsanity.com/paper/2603.02775/) (2026)
- [When Verification Hurts: Asymmetric Effects of Multi-Agent Feedback in Logic Proof Tutoring](https://mlsanity.com/paper/2603.27076/) (2026)
- [DeepTutor: Towards Agentic Personalized Tutoring](https://mlsanity.com/paper/2604.26962/) (2026)
- [Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialogues](https://mlsanity.com/paper/2605.01097/) (2026)
- [Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors](https://mlsanity.com/paper/2606.16206/) (2026)
- [Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work](https://mlsanity.com/paper/2504.00843/) (2025)
- [Pedagogy-R1: Pedagogically-Aligned Reasoning Model with Balanced Educational Benchmark](https://mlsanity.com/paper/2505.18467/) (2025)
- [MathBuddy: A Multimodal System for Affective Math Tutoring](https://mlsanity.com/paper/2508.19993/) (2025)
- [Reasoning Trajectories for Socratic Debugging of Student Code: From Misconceptions to Contradictions and Updated Beliefs](https://mlsanity.com/paper/2511.00371/) (2025)
- [AITutor-EvalKit: Exploring the Capabilities of AI Tutors](https://mlsanity.com/paper/2512.03688/) (2025)
- [The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness](https://mlsanity.com/paper/2605.05648/) (2026)
- [PEARL: Training Socratic Tutors with Pedagogically Aligned Reinforcement Learning](https://mlsanity.com/paper/2605.29582/) (2026)
- [Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments](https://mlsanity.com/paper/2606.15766/) (2026)
- [Uncertainty-Aware Generation and Decision-Making Under Ambiguity](https://mlsanity.com/paper/2606.30578/) (2026)
- [EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners](https://mlsanity.com/paper/2608.03206/) (2026)
- [Findings of the First Teaching Monster Challenge: A Benchmark of Pedagogical Content Knowledge in AI Agents](https://mlsanity.com/paper/2608.08852/) (2026)
- [CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks](https://mlsanity.com/paper/2608.18554/) (2026)
- [StudentSim: Training LLM-based Student Simulators](https://mlsanity.com/paper/2609.01591/) (2026)

## Abstract (arXiv)

Evaluating the pedagogical capabilities of AI-based tutoring models is critical for making guided progress in the field. Yet, we lack a reliable, easy-to-use, and simple-to-run evaluation that reflects the pedagogical abilities of models. To fill this gap, we present MathTutorBench, an open-source benchmark for holistic tutoring model evaluation. MathTutorBench contains a collection of datasets and metrics that broadly cover tutor abilities as defined by learning sciences research in dialog-based teaching. To score the pedagogical quality of open-ended teacher responses, we train a reward model and show it can discriminate expert from novice teacher responses with high accuracy. We evaluate a wide set of closed- and open-weight models on MathTutorBench and find that subject expertise, indicated by solving ability, does not immediately translate to good teaching. Rather, pedagogy and subject expertise appear to form a trade-off that is navigated by the degree of tutoring specialization of the model. Furthermore, tutoring appears to become more challenging in longer dialogs, where simpler questioning strategies begin to fail. We release the benchmark, code, and leaderboard openly to enable rapid benchmarking of future models.