# An Integrated Machine Learning and Hierarchical Variance Decomposition Pipeline for Student Performance Prediction and Metacognitive Calibration on Multi-Signal Telemetry

arXiv: https://arxiv.org/abs/2606.28881 · HTML: https://mlsanity.com/paper/2606.28881/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 28 references that are in the map.
- First posted on arXiv 27 Jun 2026.

## Builds on (most cited first, 6 of 6)

- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [BKT-LSTM: Efficient Student Modeling for knowledge tracing and student performance prediction](https://mlsanity.com/paper/2012.12218/) (2020)
- [Explainable Student Performance Prediction With Personalized Attention for Explaining Why A Student Fails](https://mlsanity.com/paper/2110.08268/) (2021)
- [Deep Learning to Predict Student Outcomes](https://mlsanity.com/paper/1905.02530/) (2019)
- [Grade prediction with course and student specific models](https://mlsanity.com/paper/1906.00792/) (2019)
- [A Predictive Model for Student Performance in Classrooms Using Student Interactions With an eTextbook](https://mlsanity.com/paper/2203.03713/) (2022)

## Abstract (arXiv)

Predicting student performance and characterizing metacognitive calibration are essential for personalization in intelligent tutoring systems. Prior research treats performance prediction, calibration error calculation, and variance decomposition as separate pipelines, preventing unified interpretation. I propose the Unified Behavioral Prediction and Calibration Analysis Pipeline (UBP-CAP), an integrated framework processing student pre-execution behavioral telemetry through three linked modules: (1) a LightGBM classifier with SHAP for binary correctness prediction, (2) formal calibration metrics (ECE, MCE, and Brier score decomposition) to evaluate metacognitive alignment, and (3) a crossed Generalized Linear Mixed-Effects Model (GLMM) for decomposing calibration deviations. I introduce the Predictive-Explanatory Divergence Index (PEDI), which quantifies structural divergence between predictive and explanatory feature profiles. Evaluated on 1,195 interaction records (27 students, 45 tasks), Logistic Regression achieves AUC-ROC = 0.903, outperforming LightGBM (0.878). Student naive ECE (0.109) significantly exceeds model ECE (0.068), confirming systematic miscalibration. The crossed GLMM yields ICCStudent = 0.123, showing calibration is situational rather than dispositional. PEDIcos = 0.081 (p = 0.327) indicates structural alignment between prediction and explanation on shared behavioral features.