# The Invisible Mentor: Inferring User Actions from Screen Recordings to Recommend Better Workflows

arXiv: https://arxiv.org/abs/2509.26557 · HTML: https://mlsanity.com/paper/2509.26557/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2025.

## Built on by (most cited first, 3 of 3)

- [Fix the Mind, Not the Move: Interpretable AI Assistance via Knowledge-Gap Localization](https://mlsanity.com/paper/2606.05602/) (2026)
- [Plans Work in Mysterious Ways: Evaluating a Plan Mode for Spreadsheet Agents](https://mlsanity.com/paper/2607.23670/) (2026)
- [Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows](https://mlsanity.com/paper/2609.27327/) (2026)

## Abstract (arXiv)

Many users struggle to notice when a more efficient workflow exists in feature-rich tools like Excel. Existing AI assistants offer help only after users describe their goals or problems, which can be effortful and imprecise. We present InvisibleMentor, a system that turns screen recordings of task completion into vision-grounded reflections on tasks. It detects issues such as repetitive edits and recommends more efficient alternatives based on observed behavior. Unlike prior systems that rely on logs, APIs, or user prompts, InvisibleMentor operates directly on screen recordings. It uses a two-stage pipeline: a vision-language model reconstructs actions and context, and a language model generates structured, high-fidelity suggestions. In evaluation, InvisibleMentor accurately identified inefficient workflows, and participants found its suggestions more actionable, tailored, and more helpful for learning and improvement compared to a prompt-based spreadsheet assistant.