# AutoCue: Multimodal LLM-Assisted Externalization of Implicit Inputs as Instructional Visual Cues in Screencast Tutorials

arXiv: https://arxiv.org/abs/2608.04910 · HTML: https://mlsanity.com/paper/2608.04910/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 6 of its 79 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 6 of 6)

- [A Closer Look into Recent Video-based Learning Research: A Comprehensive Review of Video Characteristics, Tools, Technologies, and Learning Effectiveness](https://mlsanity.com/paper/2301.13617/) (2023)
- [TutoAI: A Cross-domain Framework for AI-assisted Mixed-media Tutorial Creation on Physical Tasks](https://mlsanity.com/paper/2403.08049/) (2024)
- [VideoMix: Aggregating How-To Videos for Task-Oriented Learning](https://mlsanity.com/paper/2503.21130/) (2025)
- [NoteIt: A System Converting Instructional Videos to Interactable Notes Through Multimodal Video Understanding](https://mlsanity.com/paper/2508.14395/) (2025)
- [AQuA: Automated Question-Answering in Software Tutorial Videos with Visual Anchors](https://mlsanity.com/paper/2403.05213/) (2024)
- ["My toxic trait is thinking I'll remember this": gaps in the learner experience of video tutorials for feature-rich software](https://mlsanity.com/paper/2404.07114/) (2024)

## Abstract (arXiv)

Tutorial videos are widely used for learning feature-rich software, yet following screencast tutorials often breaks down in practice. Through a survey and contextual inquiry, we found that learners frequently rewind or get stuck because critical input information, especially mouse actions and keyboard-modified operations, is often implicit or missing in tutorials without input metadata. To address this problem, we present AutoCue, a multimodal LLM-assisted, human-in-the-loop tutorial augmentation pipeline for externalizing implicit inputs as instructional visual cues. AutoCue integrates frame-to-frame visual changes, narration signals, and operation guidance from official software manuals to infer likely mouse and key-modifier actions, then produces aligned cue layers and editable artifacts for human refinement. Grounded in multimedia learning and cognitive load theory, we further develop a visual cue grammar for representing mouse, keyboard, and combined inputs in software-learning tutorials. We instantiate and evaluate AutoCue in Autodesk Maya, focusing automatic inference on selected UI-mediated interactions with observable visual or textual feedback while supporting more ambiguous state changes through editable authoring artifacts. In a between-subjects study with 24 participants, the AutoCue-augmented tutorial reduced task completion time and interaction breakdowns and showed improved learner-reported experience.