# Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System

arXiv: https://arxiv.org/abs/2404.17460 · HTML: https://mlsanity.com/paper/2404.17460/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 37 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 3 of 3)

- [Learning gain differences between ChatGPT and human tutor generated algebra hints](https://mlsanity.com/paper/2302.06871/) (2023)
- [Assessing the Quality of Multiple-Choice Questions Using GPT-4 and Rule-Based Methods](https://mlsanity.com/paper/2307.08161/) (2023)
- [Evaluating ChatGPT's Decimal Skills and Feedback Generation in a Digital Learning Game](https://mlsanity.com/paper/2306.16639/) (2023)

## Built on by (most cited first, 26 of 26)

- [MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors](https://mlsanity.com/paper/2502.18940/) (2025)
- [Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues](https://mlsanity.com/paper/2503.06424/) (2025)
- [Measuring Agents in Production](https://mlsanity.com/paper/2512.04123/) (2025)
- [Simulated Students in Tutoring Dialogues: Substance or Illusion?](https://mlsanity.com/paper/2601.04025/) (2026)
- [TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students](https://mlsanity.com/paper/2505.01563/) (2025)
- [SPL: A Socratic Playground for Learning Powered by Large Language Model](https://mlsanity.com/paper/2406.13919/) (2024)
- [Do LLMs Make Mistakes Like Students? Exploring Natural Alignment between Language Models and Human Error Patterns](https://mlsanity.com/paper/2502.15140/) (2025)
- [Position: LLMs Can be Good Tutors in English Education](https://mlsanity.com/paper/2502.05467/) (2025)
- [Pedagogy-driven Evaluation of Generative AI-powered Intelligent Tutoring Systems](https://mlsanity.com/paper/2510.22581/) (2025)
- [ELI-Why: Evaluating the Pedagogical Utility of Language Model Explanations](https://mlsanity.com/paper/2506.14200/) (2025)
- [PATS: Personality-Aware Teaching Strategies with Large Language Model Tutors](https://mlsanity.com/paper/2601.08402/) (2026)
- [Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Models](https://mlsanity.com/paper/2503.04267/) (2025)
- [Improving Student-AI Interaction Through Pedagogical Prompting: An Example in Computer Science Education](https://mlsanity.com/paper/2506.19107/) (2025)
- [Large Language Models as Students Who Think Aloud: Overly Coherent, Verbose, and Confident](https://mlsanity.com/paper/2602.01015/) (2026)
- [Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialogues](https://mlsanity.com/paper/2605.01097/) (2026)
- [AERA Chat: An Interactive Platform for Automated Explainable Student Answer Assessment](https://mlsanity.com/paper/2410.09507/) (2024)
- [From Test-taking to Cognitive Scaffolding: A Pedagogical Diagnostic Benchmark for LLMs on English Standardized Tests](https://mlsanity.com/paper/2505.17056/) (2025)
- [The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory](https://mlsanity.com/paper/2503.10533/) (2025)
- [SSRLBot: Designing and Developing a Large Language Model-based Agent using Socially Shared Regulated Learning](https://mlsanity.com/paper/2505.00945/) (2025)
- [PAPPL: Personalized AI-Powered Progressive Learning Platform](https://mlsanity.com/paper/2508.14109/) (2025)
- [LLM Chatbot-Creation Approaches](https://mlsanity.com/paper/2509.13326/) (2025)
- [Representation Learning to Study Temporal Dynamics in Tutorial Scaffolding](https://mlsanity.com/paper/2603.24535/) (2026)
- [On-Demand Instructional Material Providing Agent Based on MLLM for Tutoring Support](https://mlsanity.com/paper/2603.25195/) (2026)
- [Identifying High-Confidence Social Biases in LLMs for Trustworthy Conversational Tutoring Agents](https://mlsanity.com/paper/2606.01584/) (2026)
- [From Content Generation to Learning Support: Pedagogy-Guided Generative Video Tutors for STEM Learning](https://mlsanity.com/paper/2609.24083/) (2026)
- [Do LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loop](https://mlsanity.com/paper/2610.15692/) (2026)

## Abstract (arXiv)

Conversational tutoring systems (CTSs) offer learning experiences through interactions based on natural language. They are recognized for promoting cognitive engagement and improving learning outcomes, especially in reasoning tasks. Nonetheless, the cost associated with authoring CTS content is a major obstacle to widespread adoption and to research on effective instructional design. In this paper, we discuss and evaluate a novel type of CTS that leverages recent advances in large language models (LLMs) in two ways: First, the system enables AI-assisted content authoring by inducing an easily editable tutoring script automatically from a lesson text. Second, the system automates the script orchestration in a learning-by-teaching format via two LLM-based agents (Ruffle&Riley) acting as a student and a professor. The system allows for free-form conversations that follow the ITS-typical inner and outer loop structure. We evaluate Ruffle&Riley's ability to support biology lessons in two between-subject online user studies (N = 200) comparing the system to simpler QA chatbots and reading activity. Analyzing system usage patterns, pre/post-test scores and user experience surveys, we find that Ruffle&Riley users report high levels of engagement, understanding and perceive the offered support as helpful. Even though Ruffle&Riley users require more time to complete the activity, we did not find significant differences in short-term learning gains over the reading activity. Our system architecture and user study provide various insights for designers of future CTSs. We further open-source our system to support ongoing research on effective instructional design of LLM-based learning technologies.