# Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise

arXiv: https://arxiv.org/abs/2410.03017 · HTML: https://mlsanity.com/paper/2410.03017/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 103 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [Prompting Is Programming: A Query Language for Large Language Models](https://mlsanity.com/paper/2212.06094/) (2022)
- [Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach](https://mlsanity.com/paper/2407.12687/) (2024)
- [Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes](https://mlsanity.com/paper/2310.10648/) (2023)
- [MEGAnno+: A Human-LLM Collaborative Annotation System](https://mlsanity.com/paper/2402.18050/) (2024)
- [Wordcraft: a Human-AI Collaborative Editor for Story Writing](https://mlsanity.com/paper/2107.07430/) (2021)
- [Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction](https://mlsanity.com/paper/2306.03090/) (2023)
- [The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters Exam Performances](https://mlsanity.com/paper/2407.09975/) (2024)
- [How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses](https://mlsanity.com/paper/2405.00970/) (2024)
- [Edu-ConvoKit: An Open-Source Library for Education Conversation Data](https://mlsanity.com/paper/2402.05111/) (2024)

## Built on by (most cited first, 30 of 50)

- [Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues](https://mlsanity.com/paper/2503.06424/) (2025)
- [LearnLM: Improving Gemini for Learning](https://mlsanity.com/paper/2412.16429/) (2024)
- [On Benchmarking Human-Like Intelligence in Machines](https://mlsanity.com/paper/2502.20502/) (2025)
- [Data for Mathematical Copilots: Better Ways of Presenting Proofs for Machine Learning](https://mlsanity.com/paper/2412.15184/) (2024)
- [Evaluating Gemini in an arena for learning](https://mlsanity.com/paper/2505.24477/) (2025)
- [HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants](https://mlsanity.com/paper/2509.08494/) (2025)
- [Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passes](https://mlsanity.com/paper/2410.16930/) (2024)
- [Effective Red-Teaming of Policy-Adherent Agents](https://mlsanity.com/paper/2506.09600/) (2025)
- [What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Text](https://mlsanity.com/paper/2503.04804/) (2025)
- [TeachLM: Post-Training LLMs for Education Using Authentic Learning Data](https://mlsanity.com/paper/2510.05087/) (2025)
- [Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)](https://mlsanity.com/paper/2501.13957/) (2025)
- [Belief in the Machine: Investigating Epistemological Blind Spots of Language Models](https://mlsanity.com/paper/2410.21195/) (2024)
- [CoDAE: Adapting Large Language Models for Education via Chain-of-Thought Data Augmentation](https://mlsanity.com/paper/2508.08386/) (2025)
- [Pedagogy-driven Evaluation of Generative AI-powered Intelligent Tutoring Systems](https://mlsanity.com/paper/2510.22581/) (2025)
- [Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions](https://mlsanity.com/paper/2601.15353/) (2026)
- [ComBodied Agents: a New Paradigm of Human-Centric Agentic AI](https://mlsanity.com/paper/2608.10915/) (2026)
- [When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaboration](https://mlsanity.com/paper/2506.05579/) (2025)
- [LeanTutor: Towards a Verified AI Mathematical Proof Tutor](https://mlsanity.com/paper/2506.08321/) (2025)
- [SEFL: A Framework for Generating Synthetic Educational Assignment Feedback with LLM Agents](https://mlsanity.com/paper/2502.12927/) (2025)
- [Leveraging LLMs to Assess Tutor Moves in Real-Life Dialogues: A Feasibility Study](https://mlsanity.com/paper/2506.17410/) (2025)
- [AI tutoring can safely and effectively support students: An exploratory RCT in UK classrooms](https://mlsanity.com/paper/2512.23633/) (2025)
- [Who Am I? History-Aware Profiles for Student Simulation in Tutoring Dialogues](https://mlsanity.com/paper/2605.30051/) (2026)
- [MedSimAI: Simulation and Formative Feedback Generation to Enhance Deliberate Practice in Medical Education](https://mlsanity.com/paper/2503.05793/) (2025)
- [How well do Large Language Models Recognize Instructional Moves? Establishing Baselines for Foundation Models in Educational Discourse](https://mlsanity.com/paper/2512.19903/) (2025)
- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)
- [Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors](https://mlsanity.com/paper/2606.16206/) (2026)
- [Shaping AI's Impact on Billions of Lives](https://mlsanity.com/paper/2412.02730/) (2024)
- [Speaking the Right Language: The Impact of Expertise Alignment in User-AI Interactions](https://mlsanity.com/paper/2502.18685/) (2025)
- [AITutor-EvalKit: Exploring the Capabilities of AI Tutors](https://mlsanity.com/paper/2512.03688/) (2025)
- [AI-Mediated Feedback Improves Student Revisions: A Randomized Trial with FeedbackWriter in a Large Undergraduate Course](https://mlsanity.com/paper/2602.16820/) (2026)

## Abstract (arXiv)

Generative AI, particularly Language Models (LMs), has the potential to transform real-world domains with societal impact, particularly where access to experts is limited. For example, in education, training novice educators with expert guidance is important for effectiveness but expensive, creating significant barriers to improving education quality at scale. This challenge disproportionately harms students from under-served communities, who stand to gain the most from high-quality education. We introduce Tutor CoPilot, a novel Human-AI approach that leverages a model of expert thinking to provide expert-like guidance to tutors as they tutor. This study is the first randomized controlled trial of a Human-AI system in live tutoring, involving 900 tutors and 1,800 K-12 students from historically under-served communities. Following a preregistered analysis plan, we find that students working with tutors that have access to Tutor CoPilot are 4 percentage points (p.p.) more likely to master topics (p<0.01). Notably, students of lower-rated tutors experienced the greatest benefit, improving mastery by 9 p.p. We find that Tutor CoPilot costs only 20 per-tutor annually. We analyze 550,000+ messages using classifiers to identify pedagogical strategies, and find that tutors with access to Tutor CoPilot are more likely to use high-quality strategies to foster student understanding (e.g., asking guiding questions) and less likely to give away the answer to the student. Tutor interviews highlight how Tutor CoPilot's guidance helps tutors to respond to student needs, though they flag issues in Tutor CoPilot, such as generating suggestions that are not grade-level appropriate. Altogether, our study of Tutor CoPilot demonstrates how Human-AI systems can scale expertise in real-world domains, bridge gaps in skills and create a future where high-quality education is accessible to all students.