# Towards Interpretable Deep Learning Models for Knowledge Tracing

arXiv: https://arxiv.org/abs/2005.06139 · HTML: https://mlsanity.com/paper/2005.06139/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2020.

## Built on by (most cited first, 13 of 13)

- [Improving Interpretability of Deep Sequential Knowledge Tracing Models with Question-centric Cognitive Representations](https://mlsanity.com/paper/2302.06885/) (2023)
- [A Survey of Knowledge Tracing: Models, Variants, and Applications](https://mlsanity.com/paper/2105.15106/) (2021)
- [Evaluating the Explainers: Black-Box Explainable Machine Learning for Student Success Prediction in MOOCs](https://mlsanity.com/paper/2207.00551/) (2022)
- [The future of human-centric eXplainable Artificial Intelligence (XAI) is not post-hoc explanations](https://mlsanity.com/paper/2307.00364/) (2023)
- [A Question-centric Multi-experts Contrastive Learning Framework for Improving the Accuracy and Interpretability of Deep Sequential Knowledge Tracing Models](https://mlsanity.com/paper/2403.07322/) (2024)
- [Interpretable Knowledge Tracing via Response Influence-based Counterfactual Reasoning](https://mlsanity.com/paper/2312.10045/) (2023)
- [Tracing the Data Trail: A Survey of Data Provenance, Transparency and Traceability in LLMs](https://mlsanity.com/paper/2601.14311/) (2026)
- [Interpreting Deep Knowledge Tracing Model on EdNet Dataset](https://mlsanity.com/paper/2111.00419/) (2021)
- [A Probabilistic Generative Model for Tracking Multi-Knowledge Concept Mastery Probability](https://mlsanity.com/paper/2302.08673/) (2023)
- [UIILD: A Unified Interpretable Intelligent Learning Diagnosis Framework for Intelligent Tutoring Systems](https://mlsanity.com/paper/2207.03122/) (2022)
- [Does Interpretability of Knowledge Tracing Models Support Teacher Decision Making?](https://mlsanity.com/paper/2511.02718/) (2025)
- [RegKT: Interpretable and Robust Deep Knowledge Tracing With IRT-Regularizer](https://mlsanity.com/paper/2609.21791/) (2026)
- [Stable and Faithful Explanations for Knowledge Tracing](https://mlsanity.com/paper/2609.28502/) (2026)

## Abstract (arXiv)

As an important technique for modeling the knowledge states of learners, the traditional knowledge tracing (KT) models have been widely used to support intelligent tutoring systems and MOOC platforms. Driven by the fast advancements of deep learning techniques, deep neural network has been recently adopted to design new KT models for achieving better prediction performance. However, the lack of interpretability of these models has painfully impeded their practical applications, as their outputs and working mechanisms suffer from the intransparent decision process and complex inner structures. We thus propose to adopt the post-hoc method to tackle the interpretability issue for deep learning based knowledge tracing (DLKT) models. Specifically, we focus on applying the layer-wise relevance propagation (LRP) method to interpret RNN-based DLKT model by backpropagating the relevance from the model's output layer to its input layer. The experiment results show the feasibility using the LRP method for interpreting the DLKT model's predictions, and partially validate the computed relevance scores from both question level and concept level. We believe it can be a solid step towards fully interpreting the DLKT models and promote their practical applications in the education domain.