# From Heuristics to Analytics: Forecasting Effort and Progress in Online Learning

arXiv: https://arxiv.org/abs/2605.12788 · HTML: https://mlsanity.com/paper/2605.12788/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 52 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 5 of 5)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [An Integrated Platform for Studying Learning with Intelligent Tutoring Systems: CTAT+TutorShop](https://mlsanity.com/paper/2502.10395/) (2025)
- [Starting Seatwork Earlier as a Valid Measure of Student Engagement](https://mlsanity.com/paper/2505.13341/) (2025)
- [How Learner Control and Explainable Learning Analytics on Skill Mastery Shape Student Desires to Finish and Avoid Loss in Tutored Practice](https://mlsanity.com/paper/2412.10568/) (2024)
- [Optimizing Mastery Learning by Fast-Forwarding Over-Practice Steps](https://mlsanity.com/paper/2506.17577/) (2025)

## Abstract (arXiv)

Sustained effort is essential for realizing the benefits of intelligent tutoring systems (ITS), yet many learners disengage or underuse available practice time. We introduce engagement forecasting as a supervised prediction task based on ITS logs, targeting two outcomes central to effort and learning progress: minutes practiced per week and new skills mastered per week. Using interaction log data from 425 middle-school students over a school year, we benchmark fifteen predictors including regressions, decision trees, and neural networks. We show that these feature-based models reduce mean absolute error (MAE) by 22-33% relative to heuristic baselines, including fixed-percentile rules adapted from prior work in other behavioral domains. We find that percentile heuristics systematically overpredict, whereas feature-based models better track student practice trajectories across weeks. To support explainability, we analyze feature importance and ablations, revealing target-specific patterns: effort forecasting is driven mainly by recent activity features, while progress forecasting depends more on learner-state and content difficulty signals. Finally, in a semi-structured user interview case study with eight college tutors, we examine how tutors reasoned about system-generated predictive features when setting goals with students. We find that tutors reasoned differently about effort versus progress goals in ways that mirror our pattern analysis. Together, these results establish a reproducible benchmark for forecasting weekly effort and learning progress in ITS. By making patterns of sustained effort and progress visible at a weekly timescale, engagement forecasting offers a foundation for supporting tutor-learner goal setting and timely instructional decisions.