# Simulated Students in Tutoring Dialogues: Substance or Illusion?

arXiv: https://arxiv.org/abs/2601.04025 · HTML: https://mlsanity.com/paper/2601.04025/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 75 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Builds on (most cited first, 30 of 36)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems](https://mlsanity.com/paper/2305.14536/) (2023)
- [Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation](https://mlsanity.com/paper/2405.11591/) (2024)
- [SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?](https://mlsanity.com/paper/2510.05444/) (2025)
- [CLASS: A Design Framework for building Intelligent Tutoring Systems based on Learning Science principles](https://mlsanity.com/paper/2305.13272/) (2023)
- [Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs](https://mlsanity.com/paper/2409.16490/) (2024)
- [Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues](https://mlsanity.com/paper/2503.06424/) (2025)
- [Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors](https://mlsanity.com/paper/2407.09136/) (2024)
- [Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education](https://mlsanity.com/paper/2309.14534/) (2023)
- [PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactions](https://mlsanity.com/paper/2505.17818/) (2025)
- [Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System](https://mlsanity.com/paper/2404.17460/) (2024)
- [Book2Dial: Generating Teacher-Student Interactions from Textbooks for Cost-Effective Development of Educational Chatbots](https://mlsanity.com/paper/2403.03307/) (2024)
- [SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction](https://mlsanity.com/paper/2507.05129/) (2025)
- [Training Turn-by-Turn Verifiers for Dialogue Tutoring Agents: The Curious Case of LLMs as Your Coding Tutors](https://mlsanity.com/paper/2502.13311/) (2025)
- [Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors](https://mlsanity.com/paper/2510.11502/) (2025)
- [Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models](https://mlsanity.com/paper/2407.15645/) (2024)
- [LLM-based Cognitive Models of Students with Misconceptions](https://mlsanity.com/paper/2410.12294/) (2024)
- [Evaluating and Optimizing Educational Content with Large Language Model Judgments](https://mlsanity.com/paper/2403.02795/) (2024)
- [Toward In-Context Teaching: Adapting Examples to Students' Misconceptions](https://mlsanity.com/paper/2405.04495/) (2024)
- [TutorUp: What If Your Students Were Simulated? Training Tutors to Address Engagement Challenges in Online Learning](https://mlsanity.com/paper/2502.16178/) (2025)
- [PIIvot: A Lightweight NLP Anonymization Framework for Question-Anchored Tutoring Dialogues](https://mlsanity.com/paper/2505.16931/) (2025)
- [ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulation](https://mlsanity.com/paper/2509.21730/) (2025)
- [TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students](https://mlsanity.com/paper/2505.01563/) (2025)

## Built on by (most cited first, 18 of 18)

- [Letting Tutor Personas Speak Up for LLMs: Learning Steering Vectors from Dialogue via Preference Optimization](https://mlsanity.com/paper/2602.07639/) (2026)
- [Simulating Novice Students Using Machine Unlearning and Relearning in Large Language Models](https://mlsanity.com/paper/2603.26142/) (2026)
- [Who Am I? History-Aware Profiles for Student Simulation in Tutoring Dialogues](https://mlsanity.com/paper/2605.30051/) (2026)
- [DeepTutor: Towards Agentic Personalized Tutoring](https://mlsanity.com/paper/2604.26962/) (2026)
- [Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialogues](https://mlsanity.com/paper/2605.01097/) (2026)
- [Rethinking LLM-Judged Helpfulness as a Pedagogy Signal: A Pre-Registered Audit Across Tutor Models](https://mlsanity.com/paper/2607.28128/) (2026)
- [EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners](https://mlsanity.com/paper/2608.03206/) (2026)
- [KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn](https://mlsanity.com/paper/2608.17150/) (2026)
- [StudentSim: Training LLM-based Student Simulators](https://mlsanity.com/paper/2609.01591/) (2026)
- [Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators](https://mlsanity.com/paper/2605.12748/) (2026)
- [Beyond Direct Answering: Aligning Educational LLMs as Socratic Guides via Heuristic Reinforcement Learning](https://mlsanity.com/paper/2607.22996/) (2026)
- [Auditable Release Control for Pedagogical Leakage in LLM Tutors](https://mlsanity.com/paper/2608.00515/) (2026)
- [Simulating Disengaged Students to Evaluate LLM-based Tutors](https://mlsanity.com/paper/2609.12331/) (2026)
- [EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics](https://mlsanity.com/paper/2609.12949/) (2026)
- [From Learner Behavior to Reusable Skills for Effective and Efficient Learner Simulation](https://mlsanity.com/paper/2609.37157/) (2026)
- [Sherpa: Teaching LLMs to Teach Adaptively](https://mlsanity.com/paper/2610.08778/) (2026)
- [Do LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loop](https://mlsanity.com/paper/2610.15692/) (2026)
- [A Unified Response Framework for Questions and Tutoring Dialogues](https://mlsanity.com/paper/2610.19532/) (2026)

## Abstract (arXiv)

Advances in large language models (LLMs) enable many new innovations in education. However, evaluating the effectiveness of new technology requires real students, which is time-consuming and hard to scale up. Therefore, many recent works on LLM-powered tutoring solutions have used simulated students for both training and evaluation, often via simple prompting. Surprisingly, little work has been done to ensure or even measure the quality of simulated students. In this work, we formally define the student simulation task, propose a set of evaluation metrics that span linguistic, behavioral, and cognitive aspects, and benchmark a wide range of student simulation methods on these metrics. We experiment on a real-world math tutoring dialogue dataset, where both automated and human evaluation results show that prompting strategies for student simulation perform poorly; supervised fine-tuning and preference optimization yield much better but still limited performance, motivating future work on this challenging task.