# LSTM-SAKT: LSTM-Encoded SAKT-like Transformer for Knowledge Tracing

arXiv: https://arxiv.org/abs/2102.00845 · HTML: https://mlsanity.com/paper/2102.00845/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 11 references that are in the map.
- First posted on arXiv 28 Jan 2021.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [A Self-Attentive model for Knowledge Tracing](https://mlsanity.com/paper/1907.06837/) (2019)
- [EdNet: A Large-Scale Hierarchical Dataset in Education](https://mlsanity.com/paper/1912.03072/) (2019)
- [Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing](https://mlsanity.com/paper/2002.07033/) (2020)
- [SAINT+: Integrating Temporal Features for EdNet Correctness Prediction](https://mlsanity.com/paper/2010.12042/) (2020)

## Built on by (most cited first, 5 of 5)

- [Empirical Evaluation of Deep Learning Models for Knowledge Tracing: Of Hyperparameters and Metrics on Performance and Replicability](https://mlsanity.com/paper/2112.15072/) (2021)
- [Addressing Label Leakage in Knowledge Tracing Models](https://mlsanity.com/paper/2403.15304/) (2024)
- [Multi-granulariy Time-based Transformer for Knowledge Tracing](https://mlsanity.com/paper/2304.05257/) (2023)
- [Improved Sales Forecasting using Trend and Seasonality Decomposition with LightGBM](https://mlsanity.com/paper/2305.17201/) (2023)
- [Enhancing Knowledge Tracing through Leakage-Free and Recency-Aware Embeddings](https://mlsanity.com/paper/2508.17092/) (2025)

## Abstract (arXiv)

This paper introduces the 2nd place solution for the Riiid! Answer Correctness Prediction in Kaggle, the world's largest data science competition website. This competition was held from October 16, 2020, to January 7, 2021, with 3395 teams and 4387 competitors. The main insights and contributions of this paper are as follows. (i) We pointed out existing Transformer-based models are suffering from a problem that the information which their query/key/value can contain is limited. To solve this problem, we proposed a method that uses LSTM to obtain query/key/value and verified its effectiveness. (ii) We pointed out 'inter-container' leakage problem, which happens in datasets where questions are sometimes served together. To solve this problem, we showed special indexing/masking techniques that are useful when using RNN-variants and Transformer. (iii) We found additional hand-crafted features are effective to overcome the limits of Transformer, which can never consider the samples older than the sequence length.