# Towards Breaking the Learning System Wall Using Multimodal Tutoring Transcriptions

arXiv: https://arxiv.org/abs/2609.36502 · HTML: https://mlsanity.com/paper/2609.36502/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 26 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 9 of 9)

- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding](https://mlsanity.com/paper/2405.13382/) (2024)
- [Improving Student Learning with Hybrid Human-AI Tutoring: A Three-Study Quasi-Experimental Investigation](https://mlsanity.com/paper/2312.11274/) (2023)
- [Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders](https://mlsanity.com/paper/2510.26027/) (2025)
- [Brief but Impactful: How Human Tutoring Interactions Shape Engagement in Online Learning](https://mlsanity.com/paper/2601.09994/) (2026)
- [Million Tutoring Moves (MTM): An Open Multimodal Dataset for the Science of Tutoring](https://mlsanity.com/paper/2605.08092/) (2026)
- [Improving Hybrid Human-AI Tutoring by Differentiating Human Tutor Roles Based on Student Needs](https://mlsanity.com/paper/2605.11155/) (2026)
- [A Case Study Reexamining the Cold-Start Problem in Knowledge Tracing Models and Implications for SafeInsights, an Education Research Infrastructure](https://mlsanity.com/paper/2606.11004/) (2026)

## Abstract (arXiv)

Past research using log data has faced the "learning system wall," whereby few methods exist for generalizing models of student learning across platforms. Increasingly, online learning is captured by richer forms of data, including dialog and video, with new affordances. An example of this is remote tutoring programs, where human tutors support students who use learning systems while video conferencing. Toward better platform-general modeling of learning, we introduce an AI-driven multimodal transcription system that processes screen-recording videos into unified screenplay-style transcripts containing audio dialogue and annotated learning log actions. We describe a planned method for temporally aligning AI-generated multimodal transcripts with MATHia learning logs and for identifying and classifying student learning processes to align with MATHia logs. Lastly, we highlight challenges and potential solutions in capturing learning processes in one system, offering initial steps towards generalizing log data across diverse systems.