# Integrating LSTM and BERT for Long-Sequence Data Analysis in Intelligent Tutoring Systems

arXiv: https://arxiv.org/abs/2405.05136 · HTML: https://mlsanity.com/paper/2405.05136/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 24 Apr 2024.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [Context-Aware Attentive Knowledge Tracing](https://mlsanity.com/paper/2007.12324/) (2020)
- [A Self-Attentive model for Knowledge Tracing](https://mlsanity.com/paper/1907.06837/) (2019)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [EdNet: A Large-Scale Hierarchical Dataset in Education](https://mlsanity.com/paper/1912.03072/) (2019)
- [ConvBERT: Improving BERT with Span-based Dynamic Convolution](https://mlsanity.com/paper/2008.02496/) (2020)
- [Knowledge Tracing: A Survey](https://mlsanity.com/paper/2201.06953/) (2022)
- [SAINT+: Integrating Temporal Features for EdNet Correctness Prediction](https://mlsanity.com/paper/2010.12042/) (2020)
- [LakhNES: Improving multi-instrumental music generation with cross-domain pre-training](https://mlsanity.com/paper/1907.04868/) (2019)
- [MonaCoBERT: Monotonic attention based ConvBERT for Knowledge Tracing](https://mlsanity.com/paper/2208.12615/) (2022)
- [Last Query Transformer RNN for knowledge tracing](https://mlsanity.com/paper/2102.05038/) (2021)

## Built on by (most cited first, 2 of 2)

- [LLM-KT: Aligning Large Language Models with Knowledge Tracing using a Plug-and-Play Instruction](https://mlsanity.com/paper/2502.02945/) (2025)
- [TutorLoop: Regulating Student Learning Behaviors via Sensor-in-the-Loop Generative Feedback](https://mlsanity.com/paper/2610.09400/) (2026)

## Abstract (arXiv)

The field of Knowledge Tracing aims to understand how students learn and master knowledge over time by analyzing their historical behaviour data. To achieve this goal, many researchers have proposed Knowledge Tracing models that use data from Intelligent Tutoring Systems to predict students' subsequent actions. However, with the development of Intelligent Tutoring Systems, large-scale datasets containing long-sequence data began to emerge. Recent deep learning based Knowledge Tracing models face obstacles such as low efficiency, low accuracy, and low interpretability when dealing with large-scale datasets containing long-sequence data. To address these issues and promote the sustainable development of Intelligent Tutoring Systems, we propose a LSTM BERT-based Knowledge Tracing model for long sequence data processing, namely LBKT, which uses a BERT-based architecture with a Rasch model-based embeddings block to deal with different difficulty levels information and an LSTM block to process the sequential characteristic in students' actions. LBKT achieves the best performance on most benchmark datasets on the metrics of ACC and AUC. Additionally, an ablation study is conducted to analyse the impact of each component of LBKT's overall performance. Moreover, we used t-SNE as the visualisation tool to demonstrate the model's embedding strategy. The results indicate that LBKT is faster, more interpretable, and has a lower memory cost than the traditional deep learning based Knowledge Tracing methods.