# Improving Low-Resource Knowledge Tracing Tasks by Supervised Pre-training and Importance Mechanism Fine-tuning

arXiv: https://arxiv.org/abs/2403.06725 · HTML: https://mlsanity.com/paper/2403.06725/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 62 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 30 of 36)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [Continual Pre-training of Language Models](https://mlsanity.com/paper/2302.03241/) (2023)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [Context-Aware Attentive Knowledge Tracing](https://mlsanity.com/paper/2007.12324/) (2020)
- [A Self-Attentive model for Knowledge Tracing](https://mlsanity.com/paper/1907.06837/) (2019)
- [Dynamic Key-Value Memory Networks for Knowledge Tracing](https://mlsanity.com/paper/1611.08108/) (2016)
- [Pre-training of Graph Augmented Transformers for Medication Recommendation](https://mlsanity.com/paper/1906.00346/) (2019)
- [EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction](https://mlsanity.com/paper/1906.05658/) (2019)
- [EdNet: A Large-Scale Hierarchical Dataset in Education](https://mlsanity.com/paper/1912.03072/) (2019)
- [Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing](https://mlsanity.com/paper/2002.07033/) (2020)
- [Pre-training on high-resource speech recognition improves low-resource speech-to-text translation](https://mlsanity.com/paper/1809.01431/) (2018)
- [Instructions and Guide for Diagnostic Questions: The NeurIPS 2020 Education Challenge](https://mlsanity.com/paper/2007.12061/) (2020)
- [Knowledge Tracing with Sequential Key-Value Memory Networks](https://mlsanity.com/paper/1910.13197/) (2019)
- [Addressing Two Problems in Deep Knowledge Tracing via Prediction-Consistent Regularization](https://mlsanity.com/paper/1806.02180/) (2018)
- [Deep-IRT: Make Deep Learning Based Knowledge Tracing Explainable Using Item Response Theory](https://mlsanity.com/paper/1904.11738/) (2019)
- [simpleKT: A Simple But Tough-to-Beat Baseline for Knowledge Tracing](https://mlsanity.com/paper/2302.06881/) (2023)
- [Improved Fine-Tuning by Better Leveraging Pre-Training Data](https://mlsanity.com/paper/2111.12292/) (2021)
- [Enhancing Knowledge Tracing via Adversarial Training](https://mlsanity.com/paper/2108.04430/) (2021)
- [End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning](https://mlsanity.com/paper/1904.06508/) (2019)
- [qDKT: Question-centric Deep Knowledge Tracing](https://mlsanity.com/paper/2005.12442/) (2020)
- [Enhancing Deep Knowledge Tracing with Auxiliary Tasks](https://mlsanity.com/paper/2302.07942/) (2023)
- [Knowledge Query Network: How Knowledge Interacts with Skills](https://mlsanity.com/paper/1908.02146/) (2019)
- [Bi-CLKT: Bi-Graph Contrastive Learning based Knowledge Tracing](https://mlsanity.com/paper/2201.09020/) (2022)

## Built on by (most cited first, 3 of 3)

- [Incremental Residual Concept Bottleneck Models](https://mlsanity.com/paper/2404.08978/) (2024)
- [A Question-centric Multi-experts Contrastive Learning Framework for Improving the Accuracy and Interpretability of Deep Sequential Knowledge Tracing Models](https://mlsanity.com/paper/2403.07322/) (2024)
- [Balancing Speciality and Versatility: A Coarse to Fine Framework for Mitigating Catastrophic Forgetting in Large Language Models](https://mlsanity.com/paper/2404.10306/) (2024)

## Abstract (arXiv)

Knowledge tracing (KT) aims to estimate student's knowledge mastery based on their historical interactions. Recently, the deep learning based KT (DLKT) approaches have achieved impressive performance in the KT task. These DLKT models heavily rely on the large number of available student interactions. However, due to various reasons such as budget constraints and privacy concerns, observed interactions are very limited in many real-world scenarios, a.k.a, low-resource KT datasets. Directly training a DLKT model on a low-resource KT dataset may lead to overfitting and it is difficult to choose the appropriate deep neural architecture. Therefore, in this paper, we propose a low-resource KT framework called LoReKT to address above challenges. Inspired by the prevalent "pre-training and fine-tuning" paradigm, we aim to learn transferable parameters and representations from rich-resource KT datasets during the pre-training stage and subsequently facilitate effective adaptation to low-resource KT datasets. Specifically, we simplify existing sophisticated DLKT model architectures with purely a stack of transformer decoders. We design an encoding mechanism to incorporate student interactions from multiple KT data sources and develop an importance mechanism to prioritize updating parameters with high importance while constraining less important ones during the fine-tuning stage. We evaluate LoReKT on six public KT datasets and experimental results demonstrate the superiority of our approach in terms of AUC and Accuracy. To encourage reproducible research, we make our data and code publicly available at https://github.com/rattlesnakey/LoReKT.