# Can Large Language Models Match Tutoring System Adaptivity? A Benchmarking Study

arXiv: https://arxiv.org/abs/2504.05570 · HTML: https://mlsanity.com/paper/2504.05570/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Builds on (most cited first, 17 of 17)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes](https://mlsanity.com/paper/2308.06921/) (2023)
- [Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance](https://mlsanity.com/paper/2412.09315/) (2024)
- [Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs](https://mlsanity.com/paper/2409.16490/) (2024)
- [Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciences](https://mlsanity.com/paper/2405.04645/) (2024)
- [RELIC: Investigating Large Language Model Responses using Self-Consistency](https://mlsanity.com/paper/2311.16842/) (2023)
- [When Text Embedding Meets Large Language Model: A Comprehensive Survey](https://mlsanity.com/paper/2412.09165/) (2024)
- [Ruffle&Riley: Towards the Automated Induction of Conversational Tutoring Systems](https://mlsanity.com/paper/2310.01420/) (2023)
- [Combining Large Language Models with Tutoring System Intelligence: A Case Study in Caregiver Homework Support](https://mlsanity.com/paper/2412.11995/) (2024)
- [How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-ended Responses](https://mlsanity.com/paper/2405.00291/) (2024)
- [Predicting Learning Performance with Large Language Models: A Study in Adult Literacy](https://mlsanity.com/paper/2403.14668/) (2024)
- [Understanding the Role of Temperature in Diverse Question Generation by GPT-4](https://mlsanity.com/paper/2404.09366/) (2024)
- [Do Tutors Learn from Equity Training and Can Generative AI Assess It?](https://mlsanity.com/paper/2412.11255/) (2024)

## Built on by (most cited first, 18 of 18)

- [Benchmarking Educational LLMs with Analytics: A Case Study on Gender Bias in Feedback](https://mlsanity.com/paper/2511.08225/) (2025)
- [Disentangling Learning from Judgment: Representation Learning for Open Response Analytics](https://mlsanity.com/paper/2512.23941/) (2025)
- [Evidence-Decision-Feedback: Theory-Driven Adaptive Scaffolding for LLM Agents](https://mlsanity.com/paper/2602.01415/) (2026)
- [Problems With Large Language Models for Learner Modelling: Why LLMs Alone Fall Short for Responsible Tutoring in K--12 Education](https://mlsanity.com/paper/2512.23036/) (2025)
- [Large Language Models as Students Who Think Aloud: Overly Coherent, Verbose, and Confident](https://mlsanity.com/paper/2602.01015/) (2026)
- [Small, Private Language Models as Teammates for Educational Assessment Design](https://mlsanity.com/paper/2605.15015/) (2026)
- [Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most](https://mlsanity.com/paper/2605.16207/) (2026)
- [Neural-Symbolic Knowledge Tracing: Injecting Educational Knowledge into Deep Learning for Responsible Learner Modelling](https://mlsanity.com/paper/2604.08263/) (2026)
- [PAPPL: Personalized AI-Powered Progressive Learning Platform](https://mlsanity.com/paper/2508.14109/) (2025)
- [Understanding Teacher Revisions of Large Language Model-Generated Feedback](https://mlsanity.com/paper/2603.27806/) (2026)
- [Physiological and Semantic Patterns in Medical Teams Using an Intelligent Tutoring System](https://mlsanity.com/paper/2603.29950/) (2026)
- [Can Vision Language Models Be Adaptive in Mathematics Education? A Learner Model-based Rubric Study](https://mlsanity.com/paper/2605.16011/) (2026)
- [Toward Measuring AI's Effects on Skill Formation: The Stock-Formation Gap](https://mlsanity.com/paper/2605.16283/) (2026)
- [Learning Engagement Assistant (LEA): Cross-Course Scalability and Classroom Evaluation of an Agentic AI Tutoring System](https://mlsanity.com/paper/2607.13370/) (2026)
- [CyberAGENTS: Structured Autonomy for Agentic Gamified Learning in Cybersecurity](https://mlsanity.com/paper/2608.07965/) (2026)
- [Methodologies for Improving the Quality of AI Tutoring in K-12 Education](https://mlsanity.com/paper/2608.11259/) (2026)
- [From Content Generation to Learning Support: Pedagogy-Guided Generative Video Tutors for STEM Learning](https://mlsanity.com/paper/2609.24083/) (2026)
- [Do LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loop](https://mlsanity.com/paper/2610.15692/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) hold promise as dynamic instructional aids. Yet, it remains unclear whether LLMs can replicate the adaptivity of intelligent tutoring systems (ITS)--where student knowledge and pedagogical strategies are explicitly modeled. We propose a prompt variation framework to assess LLM-generated instructional moves' adaptivity and pedagogical soundness across 75 real-world tutoring scenarios from an ITS. We systematically remove key context components (e.g., student errors and knowledge components) from prompts to create variations of each scenario. Three representative LLMs (Llama3-8B, Llama3-70B, and GPT-4o) generate 1,350 instructional moves. We use text embeddings and randomization tests to measure how the omission of each context feature impacts the LLMs' outputs (adaptivity) and a validated tutor-training classifier to evaluate response quality (pedagogical soundness). Surprisingly, even the best-performing model only marginally mimics the adaptivity of ITS. Specifically, Llama3-70B demonstrates statistically significant adaptivity to student errors. Although Llama3-8B's recommendations receive higher pedagogical soundness scores than the other models, it struggles with instruction-following behaviors, including output formatting. By contrast, GPT-4o reliably adheres to instructions but tends to provide overly direct feedback that diverges from effective tutoring, prompting learners with open-ended questions to gauge knowledge. Given these results, we discuss how current LLM-based tutoring is unlikely to produce learning benefits rivaling known-to-be-effective ITS tutoring. Through our open-source benchmarking code, we contribute a reproducible method for evaluating LLMs' instructional adaptivity and fidelity.