# Simulating Disengaged Students to Evaluate LLM-based Tutors

arXiv: https://arxiv.org/abs/2609.12331 · HTML: https://mlsanity.com/paper/2609.12331/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 11 Sep 2026.

## Builds on (most cited first, 9 of 9)

- [MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems](https://mlsanity.com/paper/2305.14536/) (2023)
- [EdNet: A Large-Scale Hierarchical Dataset in Education](https://mlsanity.com/paper/1912.03072/) (2019)
- [MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors](https://mlsanity.com/paper/2502.18940/) (2025)
- [Results and Insights from Diagnostic Questions: The NeurIPS 2020 Education Challenge](https://mlsanity.com/paper/2104.04034/) (2021)
- [Simulated Students in Tutoring Dialogues: Substance or Illusion?](https://mlsanity.com/paper/2601.04025/) (2026)
- [Starting Seatwork Earlier as a Valid Measure of Student Engagement](https://mlsanity.com/paper/2505.13341/) (2025)
- [Simulating Novice Students Using Machine Unlearning and Relearning in Large Language Models](https://mlsanity.com/paper/2603.26142/) (2026)
- [Who Am I? History-Aware Profiles for Student Simulation in Tutoring Dialogues](https://mlsanity.com/paper/2605.30051/) (2026)
- [The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness](https://mlsanity.com/paper/2605.05648/) (2026)

## Abstract (arXiv)

Simulated students generated by computational models provide a practical way to evaluate tutoring strategies and pedagogical approaches used by human and AI tutors. However, such simulations should account for disengaged behaviors, including gaming the system, wheel-spinning, and off-task behavior, because tutors may need different responses for different learner states. We present Disengagement-Aware Student Simulators (DAS2), a reproducible pre-deployment protocol that models five learner-engagement states: engaged, gaming, wheel-spinning, off-task, and mixed, and evaluates AI tutor performance across these states. Using ASSISTments09, two coders independently labeled 100 sampled tutoring sessions based on anonymized interaction-log summaries. They achieved 84% agreement (Cohen's kappa = 0.78), and among agreed cases, human consensus labels matched DAS2 rule-based labels in 81% of cases (kappa = 0.75). Conditioning simulations on intended learner states reduced the correctness-rate gap between simulated and authentic sessions from 0.54 to 0.20 for gaming and from 0.51 to 0.18 for wheel-spinning. Fine-tuned Qwen2.5-7B better matched authentic response-time distributions, while prompt-only GPT-4o generated more distinguishable learner states. Evaluation of five AI tutors from the Claude, Llama, Gemini, Qwen, and GPT families showed that relative rankings remained stable across learner states and interaction lengths, while absolute performance varied, revealing state-specific differences in tutor support. Human validation further showed that automated tutor evaluation does not fully align with human judgment. DAS2 provides a pre-deployment framework for evaluating how AI tutors respond to diverse learner-engagement states before deployment.