# TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students

arXiv: https://arxiv.org/abs/2505.01563 · HTML: https://mlsanity.com/paper/2505.01563/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Educational agents and math problem solving, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 51 references that are in the map.
- First posted on arXiv 2 May 2025.

## Placed in

- LLM agents and tool use / [Educational agents and math problem solving](https://mlsanity.com/agents/educational-agents-and-math-problem-solving/)

## Builds on (most cited first, 15 of 15)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciences](https://mlsanity.com/paper/2405.04645/) (2024)
- [AutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails](https://mlsanity.com/paper/2402.09216/) (2024)
- [Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System](https://mlsanity.com/paper/2404.17460/) (2024)
- [Beyond Final Answers: Evaluating Large Language Models for Math Tutoring](https://mlsanity.com/paper/2503.16460/) (2025)
- [Towards Educator-Driven Tutor Authoring: Generative AI Approaches for Creating Intelligent Tutor Interfaces](https://mlsanity.com/paper/2405.14713/) (2024)
- [AI2T: Building Trustable AI Tutors by Interactively Teaching a Self-Aware Learning Agent](https://mlsanity.com/paper/2411.17924/) (2024)
- [Apprentice Tutor Builder: A Platform For Users to Create and Personalize Intelligent Tutors](https://mlsanity.com/paper/2404.07883/) (2024)
- [Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs](https://mlsanity.com/paper/2404.18978/) (2024)
- [HTN-Based Tutors: A New Intelligent Tutoring Framework Based on Hierarchical Task Networks](https://mlsanity.com/paper/2405.14716/) (2024)
- [Evaluating Alternative Training Interventions Using Personalized Computational Models of Learning](https://mlsanity.com/paper/2408.13684/) (2024)
- [Model Human Learners: Computational Models to Guide Instructional Design](https://mlsanity.com/paper/2502.02456/) (2025)

## Built on by (most cited first, 10 of 10)

- [Simulated Students in Tutoring Dialogues: Substance or Illusion?](https://mlsanity.com/paper/2601.04025/) (2026)
- [Temperature and Persona Shape LLM Agent Consensus With Minimal Accuracy Gains in Qualitative Coding](https://mlsanity.com/paper/2507.11198/) (2025)
- [ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluation](https://mlsanity.com/paper/2507.12674/) (2025)
- [AI-Mediated Feedback Improves Student Revisions: A Randomized Trial with FeedbackWriter in a Large Undergraduate Course](https://mlsanity.com/paper/2602.16820/) (2026)
- [The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness](https://mlsanity.com/paper/2605.05648/) (2026)
- [SMTB: Fast Structure-Mapping with Tight Bounds](https://mlsanity.com/paper/2609.25508/) (2026)
- [Student Development Agent: Risk-free Simulation for Evaluating AIED Innovations](https://mlsanity.com/paper/2510.09183/) (2025)
- [Knowledge Distillation for Automated AI Tutor Evaluation](https://mlsanity.com/paper/2607.10647/) (2026)
- [AI-accelerated End-to-End Framework for Rapid Professional Upskilling](https://mlsanity.com/paper/2607.14044/) (2026)
- [EvalConvoLearn: An Open-Source Framework for Evaluating Grounded Learner Simulations in Tutoring Conversations](https://mlsanity.com/paper/2608.07497/) (2026)

## Abstract (arXiv)

Recent improvements in large language model (LLM) performance on academic benchmarks, such as MATH and GSM8K, have emboldened their use as standalone tutors and as simulations of human learning. However, these new applications require more than evaluations of final solution generation. We introduce TutorGym to evaluate these applications more directly. TutorGym is a standard interface for testing artificial intelligence (AI) agents within existing intelligent tutoring systems (ITS) that have been tested and refined in classroom studies, including Cognitive Tutors (CTAT), Apprentice Tutors, and OATutors. TutorGym is more than a simple problem-solution benchmark, it situates AI agents within the interactive interfaces of existing ITSs. At each step of problem-solving, AI agents are asked what they would do as a tutor or as a learner. As tutors, AI agents are prompted to provide tutoring support -- such as generating examples, hints, and step-level correctness feedback -- which can be evaluated directly against the adaptive step-by-step support provided by existing ITSs. As students, agents directly learn from ITS instruction, and their mistakes and learning trajectories can be compared to student data. TutorGym establishes a common framework for training and evaluating diverse AI agents, including LLMs, computational models of learning, and reinforcement learning agents, within a growing suite of learning environments. Currently, TutorGym includes 223 different tutor domains. In an initial evaluation, we find that current LLMs are poor at tutoring -- none did better than chance at labeling incorrect actions, and next-step actions were correct only ~52-70% of the time -- but they could produce remarkably human-like learning curves when trained as students with in-context learning.