# Next Token Knowledge Tracing: Exploiting Pretrained LLM Representations to Decode Student Behaviour

arXiv: https://arxiv.org/abs/2511.02599 · HTML: https://mlsanity.com/paper/2511.02599/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 4 Nov 2025.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [Context-Aware Attentive Knowledge Tracing](https://mlsanity.com/paper/2007.12324/) (2020)
- [A Self-Attentive model for Knowledge Tracing](https://mlsanity.com/paper/1907.06837/) (2019)
- [Dynamic Key-Value Memory Networks for Knowledge Tracing](https://mlsanity.com/paper/1611.08108/) (2016)
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction](https://mlsanity.com/paper/1906.05658/) (2019)
- [EdNet: A Large-Scale Hierarchical Dataset in Education](https://mlsanity.com/paper/1912.03072/) (2019)
- [Large language models surpass human experts in predicting neuroscience results](https://mlsanity.com/paper/2403.03230/) (2024)
- [Instructions and Guide for Diagnostic Questions: The NeurIPS 2020 Education Challenge](https://mlsanity.com/paper/2007.12061/) (2020)
- [RKT : Relation-Aware Self-Attention for Knowledge Tracing](https://mlsanity.com/paper/2008.12736/) (2020)
- [Understanding the Performance and Estimating the Cost of LLM Fine-Tuning](https://mlsanity.com/paper/2408.04693/) (2024)
- [Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling](https://mlsanity.com/paper/2401.14556/) (2024)
- [Language Model Can Do Knowledge Tracing: Simple but Effective Method to Integrate Language Model and Knowledge Tracing Task](https://mlsanity.com/paper/2406.02893/) (2024)
- [TrueLearn: A Family of Bayesian Algorithms to Match Lifelong Learners to Open Educational Resources](https://mlsanity.com/paper/1911.09471/) (2019)
- [CLST: Cold-Start Mitigation in Knowledge Tracing by Aligning a Generative Language Model as a Students' Knowledge Tracer](https://mlsanity.com/paper/2406.10296/) (2024)
- [A Novel Approach to Scalable and Automatic Topic-Controlled Question Generation in Education](https://mlsanity.com/paper/2501.05220/) (2025)
- [An AI-based Learning Companion Promoting Lifelong Learning Opportunities for All](https://mlsanity.com/paper/2112.01242/) (2021)
- [Loss Masking Is Not Needed in Decoder-only Transformer for Discrete-token-based ASR](https://mlsanity.com/paper/2311.04534/) (2023)
- [Watch Less and Uncover More: Could Navigation Tools Help Users Search and Explore Videos?](https://mlsanity.com/paper/2201.03408/) (2022)
- [A Toolbox for Modelling Engagement with Educational Videos](https://mlsanity.com/paper/2401.05424/) (2023)

## Built on by (most cited first, 4 of 4)

- [Catching The Correct Answer Trap: Characterising AI Tutor Blind Spots When Analysing Student Reasoning](https://mlsanity.com/paper/2605.23925/) (2026)
- [Agentic Knowledge Tracing: A Multi-Agent LLM Architecture for Stealth Assessment of Financial Literacy in Serious Games](https://mlsanity.com/paper/2606.25358/) (2026)
- [Faster, Cheaper, More Accurate: Specialised Knowledge Tracing Models Outperform LLMs](https://mlsanity.com/paper/2603.02830/) (2026)
- [The Correct Answer Trap: Pedagogically-Grounded Detection and Feedback for Hidden Misconceptions](https://mlsanity.com/paper/2606.23205/) (2026)

## Abstract (arXiv)

Modelling student knowledge is a key challenge when leveraging AI in education, with major implications for personalised learning. The Knowledge Tracing (KT) task aims to predict how students will respond to educational questions in learning environments, based on their prior interactions. Existing KT models typically use response correctness along with metadata like skill tags and timestamps, often overlooking the question text, which is an important source of pedagogical insight. This omission poses a lost opportunity while limiting predictive performance. We propose Next Token Knowledge Tracing (NTKT), a novel approach that reframes KT as a next-token prediction task using pretrained Large Language Models (LLMs). NTKT represents both student histories and question content as sequences of text, allowing LLMs to learn patterns in both behaviour and language. Our series of experiments significantly improves performance over state-of-the-art neural KT models and generalises much better to cold-start questions and users. These findings highlight the importance of question content in KT and demonstrate the benefits of leveraging pretrained representations of LLMs to model student learning more effectively.