mlsanityData to 14 Oct 2026

MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors

arXiv 2502.18940 ↗First posted 26 Feb 2025cs.CL cs.AI cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

Evaluating the pedagogical capabilities of AI-based tutoring models is critical for making guided progress in the field. Yet, we lack a reliable, easy-to-use, and simple-to-run evaluation that reflects the pedagogical abilities of models. To fill this gap, we present MathTutorBench, an open-source benchmark for holistic tutoring model evaluation. MathTutorBench contains a collection of datasets and metrics that broadly cover tutor abilities as defined by learning sciences research in dialog-based teaching. To score the pedagogical quality of open-ended teacher responses, we train a reward model and show it can discriminate expert from novice teacher responses with high accuracy. We evaluate a wide set of closed- and open-weight models on MathTutorBench and find that subject expertise, indicated by solving ability, does not immediately translate to good teaching. Rather, pedagogy and subject expertise appear to form a trade-off that is navigated by the degree of tutoring specialization of the model. Furthermore, tutoring appears to become more challenging in longer dialogs, where simpler questioning strategies begin to fail. We release the benchmark, code, and leaderboard openly to enable rapid benchmarking of future models.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 24 of its 46 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

24 of 46 references.

References · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  7. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  8. 2023MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problemscited by 131
  9. 2022The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialoguescited by 79
  10. 2024Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutorscited by 61
  11. 2022Dialog Inpainting: Turning Documents into Dialogscited by 56
  12. 2023Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakescited by 52
12 more
  1. 2023CLASS: A Design Framework for building Intelligent Tutoring Systems based on Learning Science principlescited by 48
  2. 2021Measuring Conversational Uptake: A Case Study on Student-Teacher Interactionscited by 44
  3. 2022The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcriptscited by 43
  4. 2022Automatic Generation of Socratic Subquestions for Teaching Math Word Problemscited by 43
  5. 2024Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutorscited by 41
  6. 2023Opportunities and Challenges in Neural Dialog Tutoringcited by 33
  7. 2023The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialoguescited by 27
  8. 2024Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring Systemcited by 26
  9. 2024MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactionscited by 26
  10. 2024Book2Dial: Generating Teacher-Student Interactions from Textbooks for Cost-Effective Development of Educational Chatbotscited by 25
  11. 2024Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debuggingcited by 14
  12. 2024The Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Educationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 35.

49 papers.

Sort:

By area:

  1. 2025LLM Agents for Education: Advances and Applicationscited by 59
  2. 2025PIIvot: A Lightweight NLP Anonymization Framework for Question-Anchored Tutoring Dialoguescited by 11
  3. 2025Benchmarking the Pedagogical Knowledge of Large Language Modelscited by 5
  4. 2025Pedagogy-driven Evaluation of Generative AI-powered Intelligent Tutoring Systemscited by 5
  5. 2025DualSchool: How Reliable are LLMs for Optimization Education?cited by 4
  6. 2026FeedEval: Pedagogically Aligned Evaluation of LLM-Generated Essay Feedbackcited by 4
  7. 2026EduEVAL-DB: A Role-Based Dataset for Pedagogical Risk Evaluation in Educational Explanationscited by 4
  8. 2026Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Educationcited by 3
  9. 2025ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluationcited by 2
  10. 2025Shaping Explanations: Semantic Reward Modeling with Encoder-Only Transformers for GRPOcited by 2
  11. 2025MMTutorBench: The First Multimodal Benchmark for AI Math Tutoringcited by 2
  12. 2025How well do Large Language Models Recognize Instructional Moves? Establishing Baselines for Foundation Models in Educational Discoursecited by 2
37 more
  1. 2026From Solver to Tutor: Evaluating the Pedagogical Intelligence of LLMs with KMP-Benchcited by 2
  2. 2026When Verification Hurts: Asymmetric Effects of Multi-Agent Feedback in Logic Proof Tutoringcited by 2
  3. 2026DeepTutor: Towards Agentic Personalized Tutoringcited by 2
  4. 2026Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialoguescited by 2
  5. 2026Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutorscited by 2
  6. 2025Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Workcited by 1
  7. 2025Pedagogy-R1: Pedagogically-Aligned Reasoning Model with Balanced Educational Benchmarkcited by 1
  8. 2025MathBuddy: A Multimodal System for Affective Math Tutoringcited by 1
  9. 2025Reasoning Trajectories for Socratic Debugging of Student Code: From Misconceptions to Contradictions and Updated Beliefscited by 1
  10. 2025AITutor-EvalKit: Exploring the Capabilities of AI Tutorscited by 1
  11. 2026The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectivenesscited by 1
  12. 2026PEARL: Training Socratic Tutors with Pedagogically Aligned Reinforcement Learningcited by 1
  13. 2026Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deploymentscited by 1
  14. 2026Uncertainty-Aware Generation and Decision-Making Under Ambiguitycited by 1
  15. 2026EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learnerscited by 1
  16. 2026Findings of the First Teaching Monster Challenge: A Benchmark of Pedagogical Content Knowledge in AI Agentscited by 1
  17. 2026CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Taskscited by 1
  18. 2026StudentSim: Training LLM-based Student Simulatorscited by 1
  19. 2026TeachBench: A Syllabus-Grounded Framework for Evaluating Teaching Ability in Large Language Modelscited by 0
  20. 2026ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agentscited by 0
  21. 2026Towards Reward Modeling for AI Tutors in Math Mistake Remediationcited by 0
  22. 2026AI Meets Mathematics Education: A Case Study on Supporting an Instructor in a Large Mathematics Class with Context-Aware AIcited by 0
  23. 2026Fighting AI with AI: AI-Agent Augmented DNS Blocking of LLM Services during Student Evaluationscited by 0
  24. 2026BLADE: Better Language Answers through Dialogue and Explanationscited by 0
  25. 2026Are Agents Ready to Teach? A Multi-Stage Benchmark for Real-World Teaching Workflowscited by 0
  26. 2026LLMs Are Already Good Tutors: Training-Free Prompt Optimization for Pedagogical Math Tutoringcited by 0
  27. 2026DigitalCoach: Communication and Grounding Gaps in Human and Agentic Computer Use Coachingcited by 0
  28. 2026Context-Masked Truncated Reasoning Audits for Answer-Key Dependence in LLM Tutorscited by 0
  29. 2026Self-Aware Recursively Self-Improving Agents for Personal Singularity: A Goal-, Scope-, Tool-, and Benchmark-Driven Multi-Agent Architecturecited by 0
  30. 2026Mitigating Scaffolding Collapse in Socratic Tutors via Representation Alignmentcited by 0
  31. 2026Auditable Release Control for Pedagogical Leakage in LLM Tutorscited by 0
  32. 2026Methodologies for Improving the Quality of AI Tutoring in K-12 Educationcited by 0
  33. 2026Simulating Disengaged Students to Evaluate LLM-based Tutorscited by 0
  34. 2026EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographicscited by 0
  35. 2026OmniEdu: Open Foundation Models for Learning and Teachingcited by 0
  36. 2026Do LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loopcited by 0
  37. 2026A Unified Response Framework for Questions and Tutoring Dialoguescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles
  2. 2026The Tutoring Effectiveness Index: Predicting LLM Math Tutor Quality from Four Conversation Signals
  3. 2026Knowledge Distillation for Automated AI Tutor Evaluation
  4. 2026Towards Pedagogically Aligned LLM Tutors for Math Mistake Remediation
  5. 2026Simulated Students in Tutoring Dialogues: Substance or Illusion?
  6. 2025MathEDU: Feedback Generation on Problem-Solving Processes for Mathematical Learning Support
  7. 2026Is Mathematical Problem-Solving Expertise in Large Language Models Associated with Assessment Performance?
  8. 2025TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students