# Last Query Transformer RNN for knowledge tracing

arXiv: https://arxiv.org/abs/2102.05038 · HTML: https://mlsanity.com/paper/2102.05038/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 9 references that are in the map.
- First posted on arXiv 10 Feb 2021.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [Context-Aware Attentive Knowledge Tracing](https://mlsanity.com/paper/2007.12324/) (2020)
- [A Self-Attentive model for Knowledge Tracing](https://mlsanity.com/paper/1907.06837/) (2019)
- [EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction](https://mlsanity.com/paper/1906.05658/) (2019)
- [EdNet: A Large-Scale Hierarchical Dataset in Education](https://mlsanity.com/paper/1912.03072/) (2019)
- [Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing](https://mlsanity.com/paper/2002.07033/) (2020)
- [SAINT+: Integrating Temporal Features for EdNet Correctness Prediction](https://mlsanity.com/paper/2010.12042/) (2020)
- [Creating A Neural Pedagogical Agent by Jointly Learning to Review and Assess](https://mlsanity.com/paper/1906.10910/) (2019)

## Built on by (most cited first, 5 of 5)

- [Integrating LSTM and BERT for Long-Sequence Data Analysis in Intelligent Tutoring Systems](https://mlsanity.com/paper/2405.05136/) (2024)
- [Multi-granulariy Time-based Transformer for Knowledge Tracing](https://mlsanity.com/paper/2304.05257/) (2023)
- [Improved Sales Forecasting using Trend and Seasonality Decomposition with LightGBM](https://mlsanity.com/paper/2305.17201/) (2023)
- [Improving Question Embeddings with Cognitive Representation Optimization for Knowledge Tracing](https://mlsanity.com/paper/2504.04121/) (2025)
- [Dynamic Programming Techniques for Enhancing Cognitive Representation in Knowledge Tracing](https://mlsanity.com/paper/2506.02949/) (2025)

## Abstract (arXiv)

This paper presents an efficient model to predict a student's answer correctness given his past learning activities. Basically, I use both transformer encoder and RNN to deal with time series input. The novel point of the model is that it only uses the last input as query in transformer encoder, instead of all sequence, which makes QK matrix multiplication in transformer Encoder to have O(L) time complexity, instead of O(L^2). It allows the model to input longer sequence. Using this model I achieved the 1st place in the 'Riiid! Answer Correctness Prediction' competition hosted on kaggle.