# EvalConvoLearn: An Open-Source Framework for Evaluating Grounded Learner Simulations in Tutoring Conversations

arXiv: https://arxiv.org/abs/2608.07497 · HTML: https://mlsanity.com/paper/2608.07497/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Builds on (most cited first, 6 of 6)

- [Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost](https://mlsanity.com/paper/2407.19825/) (2024)
- [Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs](https://mlsanity.com/paper/2409.16490/) (2024)
- [The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves](https://mlsanity.com/paper/2204.09652/) (2022)
- [TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students](https://mlsanity.com/paper/2505.01563/) (2025)
- [TeachLM: Post-Training LLMs for Education Using Authentic Learning Data](https://mlsanity.com/paper/2510.05087/) (2025)
- [Towards Valid Student Simulation with Large Language Models](https://mlsanity.com/paper/2601.05473/) (2026)

## Abstract (arXiv)

Conversational learner simulations are valuable tools for testing learning theories, evaluating instructional materials and automated tutors, or powering teachable agents. Recently, large language models (LLM) have enabled richer, more naturalistic interactions with simulated learners; however, no open framework exists for evaluating whether such simulations faithfully reproduce real learner behavior. We introduce EvalConvoLearn, an open-source framework that assesses learner simulations along two axes: learning behavior (skill-conditioned mastery outcomes) and conversational quality (talk moves, error type distributions, question rate, turn length). EvalConvoLearn measures how closely a simulated learner approximates answer distributions observed in data by grounding metrics in authentic tutoring conversation datasets, and anchoring generated tutor responses in existing tutor utterances. The framework is demonstrated on a dataset of tutoring dialogues, including results for two LLM-based learner simulations, and the published GitHub code.