# Knowing When to Defer: Selective Prediction for Responsible Knowledge Tracing

arXiv: https://arxiv.org/abs/2509.21514 · HTML: https://mlsanity.com/paper/2509.21514/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 16 of 16)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Context-Aware Attentive Knowledge Tracing](https://mlsanity.com/paper/2007.12324/) (2020)
- [A Self-Attentive model for Knowledge Tracing](https://mlsanity.com/paper/1907.06837/) (2019)
- [Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?](https://mlsanity.com/paper/2209.03302/) (2022)
- [A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement](https://mlsanity.com/paper/2204.09308/) (2022)
- [GIKT: A Graph-based Interaction Model for Knowledge Tracing](https://mlsanity.com/paper/2009.05991/) (2020)
- [Improving Knowledge Tracing via Pre-training Question Embeddings](https://mlsanity.com/paper/2012.05031/) (2020)
- [Knowledge Query Network: How Knowledge Interacts with Skills](https://mlsanity.com/paper/1908.02146/) (2019)
- [Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracing](https://mlsanity.com/paper/2410.01727/) (2024)
- [LLM-KT: Aligning Large Language Models with Knowledge Tracing using a Plug-and-Play Instruction](https://mlsanity.com/paper/2502.02945/) (2025)
- [Uncertainty-aware Knowledge Tracing](https://mlsanity.com/paper/2501.05415/) (2025)

## Abstract (arXiv)

Research on Knowledge Tracing (KT) models traditionally focuses on improving predictive accuracy. However, responsible real-world deployment requires models to know when to defer uncertain predictions to a human teacher. We introduce an intrinsic selective prediction layer for existing KT models using Monte Carlo Dropout (MC-Dropout) to quantify uncertainty. We evaluate this approach across three architectures (DKT, SAKT, and AKT) using the Eedi mathematics dataset. Abstaining on the 20% most uncertain predictions lifts accuracy by 2.3 to 3.0 percentage points, AUC by 1.9 to 2.4 percentage points and F1 by 1.4 to 4.3 percentage points without any retraining. This abstention strategy is highly targeted: the deferred set exhibits 1.45 to 1.60 times the error rate of the kept set. Furthermore, this targeting holds within every question-difficulty quartile and remains fair across student-ability levels. Importantly, MC-Dropout variance gives roughly five times the AUC lift of a calibrated two-parameter logistic (2PL) Item Response Theory (IRT) baseline as a selective-prediction signal. A variance decomposition of the model's epistemic uncertainty (BALD) reveals that the entire classical psychometric stack, comprising question difficulty, student ability, IRT-style outcome ambiguity, and historical curriculum coverage, explains less than 4% of the signal under linear modeling and at most 23% even with a non-linear regressor. This leaves 77% to 90% as architecture-specific epistemic content that MC-Dropout surfaces and simpler proxies cannot recover. Selective prediction with model-native epistemic uncertainty is therefore a necessary component of responsible KT deployment, complementary to subgroup-fairness audits and downstream classroom evaluation rather than a substitute for them.