# Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs

arXiv: https://arxiv.org/abs/2409.16490 · HTML: https://mlsanity.com/paper/2409.16490/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 70 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Builds on (most cited first, 30 of 31)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [Context-Aware Attentive Knowledge Tracing](https://mlsanity.com/paper/2007.12324/) (2020)
- [A Self-Attentive model for Knowledge Tracing](https://mlsanity.com/paper/1907.06837/) (2019)
- [Dynamic Key-Value Memory Networks for Knowledge Tracing](https://mlsanity.com/paper/1611.08108/) (2016)
- [MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems](https://mlsanity.com/paper/2305.14536/) (2023)
- [EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction](https://mlsanity.com/paper/1906.05658/) (2019)
- [Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing](https://mlsanity.com/paper/2002.07033/) (2020)
- [SAINT+: Integrating Temporal Features for EdNet Correctness Prediction](https://mlsanity.com/paper/2010.12042/) (2020)
- [pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models](https://mlsanity.com/paper/2206.11460/) (2022)
- [Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes](https://mlsanity.com/paper/2310.10648/) (2023)
- [GIKT: A Graph-based Interaction Model for Knowledge Tracing](https://mlsanity.com/paper/2009.05991/) (2020)
- [simpleKT: A Simple But Tough-to-Beat Baseline for Knowledge Tracing](https://mlsanity.com/paper/2302.06881/) (2023)
- [Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions](https://mlsanity.com/paper/2106.03873/) (2021)
- [The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts](https://mlsanity.com/paper/2211.11772/) (2022)
- [The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves](https://mlsanity.com/paper/2204.09652/) (2022)
- [How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging](https://mlsanity.com/paper/2310.05292/) (2023)
- [DAS3H: Modeling Student Learning and Forgetting for Optimally Scheduling Distributed Practice of Skills](https://mlsanity.com/paper/1905.06873/) (2019)
- [Question Generation for Adaptive Education](https://mlsanity.com/paper/2106.04262/) (2021)
- [Language Model Can Do Knowledge Tracing: Simple but Effective Method to Integrate Language Model and Knowledge Tracing Task](https://mlsanity.com/paper/2406.02893/) (2024)
- [Improving the Validity of Automatically Generated Feedback via Reinforcement Learning](https://mlsanity.com/paper/2403.01304/) (2024)
- [Adaptive and Personalized Exercise Generation for Online Language Learning](https://mlsanity.com/paper/2306.02457/) (2023)
- [CLST: Cold-Start Mitigation in Knowledge Tracing by Aligning a Generative Language Model as a Students' Knowledge Tracer](https://mlsanity.com/paper/2406.10296/) (2024)
- [pyBKT: An Accessible Python Library of Bayesian Knowledge Tracing Models](https://mlsanity.com/paper/2105.00385/) (2021)
- [Explainable Few-shot Knowledge Tracing](https://mlsanity.com/paper/2405.14391/) (2024)
- [DiVERT: Distractor Generation with Variational Errors Represented as Text for Math Multiple-choice Questions](https://mlsanity.com/paper/2406.19356/) (2024)
- [Content Knowledge Identification with Multi-Agent Large Language Models (LLMs)](https://mlsanity.com/paper/2404.07960/) (2024)
- [Knowledge Tagging System on Math Questions via LLMs with Flexible Demonstration Retriever](https://mlsanity.com/paper/2406.13885/) (2024)
- [Mathfish: Evaluating Language Model Math Reasoning via Grounding in Educational Curricula](https://mlsanity.com/paper/2408.04226/) (2024)

## Built on by (most cited first, 30 of 46)

- [LLM Agents for Education: Advances and Applications](https://mlsanity.com/paper/2503.11733/) (2025)
- [Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues](https://mlsanity.com/paper/2503.06424/) (2025)
- [SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction](https://mlsanity.com/paper/2507.05129/) (2025)
- [Training Turn-by-Turn Verifiers for Dialogue Tutoring Agents: The Curious Case of LLMs as Your Coding Tutors](https://mlsanity.com/paper/2502.13311/) (2025)
- [Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors](https://mlsanity.com/paper/2510.11502/) (2025)
- [Can Large Language Models Match Tutoring System Adaptivity? A Benchmarking Study](https://mlsanity.com/paper/2504.05570/) (2025)
- [Simulated Students in Tutoring Dialogues: Substance or Illusion?](https://mlsanity.com/paper/2601.04025/) (2026)
- [A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Education](https://mlsanity.com/paper/2410.21287/) (2024)
- [Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization](https://mlsanity.com/paper/2506.06964/) (2025)
- [Which Type of Students can LLMs Act? Investigating Authentic Simulation with Graph-based Human-AI Collaborative System](https://mlsanity.com/paper/2502.11678/) (2025)
- [Do LLMs Make Mistakes Like Students? Exploring Natural Alignment between Language Models and Human Error Patterns](https://mlsanity.com/paper/2502.15140/) (2025)
- [KASER: Knowledge-Aligned Student Error Simulator for Open-Ended Coding Tasks](https://mlsanity.com/paper/2601.06633/) (2026)
- [Position: LLMs Can be Good Tutors in English Education](https://mlsanity.com/paper/2502.05467/) (2025)
- [Automated Knowledge Component Generation for Interpretable Knowledge Tracing in Coding Problems](https://mlsanity.com/paper/2502.18632/) (2025)
- [Exploring LLMs for Predicting Tutor Strategy and Student Outcomes in Dialogues](https://mlsanity.com/paper/2507.06910/) (2025)
- [FoundationalASSIST: An Educational Dataset for Foundational Knowledge Tracing and Pedagogical Grounding of LLMs](https://mlsanity.com/paper/2602.00070/) (2026)
- [Letting Tutor Personas Speak Up for LLMs: Learning Steering Vectors from Dialogue via Preference Optimization](https://mlsanity.com/paper/2602.07639/) (2026)
- [AI Annotation Orchestration: Evaluating LLM verifiers to Improve the Quality of LLM Annotations in Learning Analytics](https://mlsanity.com/paper/2511.09785/) (2025)
- [Teaching According to Students' Aptitude: Personalized Mathematics Tutoring via Persona-, Memory-, and Forgetting-Aware LLMs](https://mlsanity.com/paper/2511.15163/) (2025)
- [Who Am I? History-Aware Profiles for Student Simulation in Tutoring Dialogues](https://mlsanity.com/paper/2605.30051/) (2026)
- [DiaCDM: Cognitive Diagnosis in Teacher-Student Dialogues using the Initiation-Response-Evaluation Framework](https://mlsanity.com/paper/2509.24821/) (2025)
- [Problems With Large Language Models for Learner Modelling: Why LLMs Alone Fall Short for Responsible Tutoring in K--12 Education](https://mlsanity.com/paper/2512.23036/) (2025)
- [Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialogues](https://mlsanity.com/paper/2605.01097/) (2026)
- [UniEDU: A Unified Language and Vision Assistant for Education Applications](https://mlsanity.com/paper/2503.20701/) (2025)
- [Is Passive Expertise-Based Personalization Enough? A Case Study in AI-Assisted Test-Taking](https://mlsanity.com/paper/2511.23376/) (2025)
- [Misconception Diagnosis From Student-Tutor Dialogue: Generate, Retrieve, Rerank](https://mlsanity.com/paper/2602.02414/) (2026)
- [Teaching Language Models How to Code Like Learners: Conversational Serialization for Student Simulation](https://mlsanity.com/paper/2604.10720/) (2026)
- [Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments](https://mlsanity.com/paper/2606.15766/) (2026)
- [Agentic Knowledge Tracing: A Multi-Agent LLM Architecture for Stealth Assessment of Financial Literacy in Serious Games](https://mlsanity.com/paper/2606.25358/) (2026)
- [KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn](https://mlsanity.com/paper/2608.17150/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have led to the development of artificial intelligence (AI)-powered tutoring chatbots, showing promise in providing broad access to high-quality personalized education. Existing works have studied how to make LLMs follow tutoring principles, but have not studied broader uses of LLMs for supporting tutoring. Up until now, tracing student knowledge and analyzing misconceptions has been difficult and time-consuming to implement for open-ended dialogue tutoring. In this work, we investigate whether LLMs can be supportive of this task: we first use LLM prompting methods to identify the knowledge components/skills involved in each dialogue turn, i.e., a tutor utterance posing a task or a student utterance that responds to it. We also evaluate whether the student responds correctly to the tutor and verify the LLM's accuracy using human expert annotations. We then apply a range of knowledge tracing (KT) methods on the resulting labeled data to track student knowledge levels over an entire dialogue. We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues. We perform extensive qualitative analyses to highlight the challenges in dialogueKT and outline multiple avenues for future work.