# Scalable and Explainable Learner-Video Interaction Prediction using Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2604.04482 · HTML: https://mlsanity.com/paper/2604.04482/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 9 of 9)

- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Meta Transfer Learning for Early Success Prediction in MOOCs](https://mlsanity.com/paper/2205.01064/) (2022)
- [RIPPLE: Concept-Based Interpretation for Raw Time Series Models in Education](https://mlsanity.com/paper/2212.01133/) (2022)
- [A Closer Look into Recent Video-based Learning Research: A Comprehensive Review of Video Characteristics, Tools, Technologies, and Learning Effectiveness](https://mlsanity.com/paper/2301.13617/) (2023)
- [Analyzing Student Strategies In Blended Courses Using Clickstream Data](https://mlsanity.com/paper/2006.00421/) (2020)
- [Your click decides your fate: Inferring Information Processing and Attrition Behavior from MOOC Video Clickstream Interactions](https://mlsanity.com/paper/1407.7131/) (2014)
- [From Formulas to Figures: How Visual Elements Impact User Interactions in Educational Videos](https://mlsanity.com/paper/2505.01753/) (2025)
- [Playback-centric visualisations of video usage using weighted interactions to guide where to watch in an educational context](https://mlsanity.com/paper/2207.12903/) (2022)

## Abstract (arXiv)

Learners' use of video controls in educational videos provides implicit signals of cognitive processing and instructional design quality, yet the lack of scalable and explainable predictive models limits instructors' ability to anticipate such behavior before deployment. We propose a scalable, interpretable pipeline for predicting population-level watching, pausing, skipping, and rewinding behavior as proxies for cognitive load from video content alone. Our approach leverages multimodal large language models (MLLMs) to compute embeddings of short video segments and trains a neural classifier to identify temporally fine-grained interaction peaks. Drawing from multimedia learning theory on instructional design for optimal cognitive load, we code features of the video segments using GPT-5 and employ them as a basis for interpreting model predictions via concept activation vectors. We evaluate our pipeline on 77 million video control events from 66 online courses. Our findings demonstrate that classifiers based on MLLM embeddings reliably predict interaction peaks, generalize to unseen academic fields, and encode interpretable, theory-relevant instructional concepts. Overall, our results show the feasibility of cost-efficient, interpretable pre-screening of educational video design and open new opportunities to empirically examine multimedia learning theory at scale.