# Generative Tutorial: Towards Live Contextualized Visual Instructions for Physical Tasks

arXiv: https://arxiv.org/abs/2609.24955 · HTML: https://mlsanity.com/paper/2609.24955/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 126 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Builds on (most cited first, 29 of 29)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [WorldScribe: Towards Context-Aware Live Visual Descriptions](https://mlsanity.com/paper/2408.06627/) (2024)
- [Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modeling](https://mlsanity.com/paper/2410.16668/) (2024)
- [WorldSmith: Iterative and Expressive Prompting for World Building with a Generative AI](https://mlsanity.com/paper/2308.13355/) (2023)
- [Vid2Coach: Transforming How-To Videos into Task Assistants](https://mlsanity.com/paper/2506.00717/) (2025)
- [PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatch](https://mlsanity.com/paper/2407.16785/) (2024)
- [Block and Detail: Scaffolding Sketch-to-Image Generation](https://mlsanity.com/paper/2402.18116/) (2024)
- [Video2Action: Reducing Human Interactions in Action Annotation of App Tutorial Videos](https://mlsanity.com/paper/2308.03252/) (2023)
- [ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions](https://mlsanity.com/paper/2412.01987/) (2024)
- [Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models](https://mlsanity.com/paper/2508.03547/) (2025)
- [HelpViz: Automatic Generation of Contextual Visual MobileTutorials from Text-Based Instructions](https://mlsanity.com/paper/2108.03356/) (2021)
- [Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook Diagrams](https://mlsanity.com/paper/2405.18614/) (2024)
- [Human I/O: Towards a Unified Approach to Detecting Situational Impairments](https://mlsanity.com/paper/2403.04008/) (2024)
- [Behavioral Indicators of Overreliance During Interaction with Conversational Language Models](https://mlsanity.com/paper/2602.11567/) (2026)
- [avaTTAR: Table Tennis Stroke Training with On-body and Detached Visualization in Augmented Reality](https://mlsanity.com/paper/2407.15373/) (2024)
- [BlendScape: Enabling End-User Customization of Video-Conferencing Environments through Generative AI](https://mlsanity.com/paper/2403.13947/) (2024)
- [From Following to Understanding: Investigating the Role of Reflective Prompts in AR-Guided Tasks to Promote Task Understanding](https://mlsanity.com/paper/2501.13258/) (2025)
- [Impact of delayed response on Wearable Cognitive Assistance](https://mlsanity.com/paper/2011.02555/) (2020)
- [PaperToPlace: Transforming Instruction Documents into Spatialized and Context-Aware Mixed Reality Experiences](https://mlsanity.com/paper/2308.13924/) (2023)
- [AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multi-modal Information Between Reality and Videos](https://mlsanity.com/paper/2507.10963/) (2025)
- [Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz](https://mlsanity.com/paper/2510.06872/) (2025)
- [Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs](https://mlsanity.com/paper/2512.20129/) (2025)
- [eXplainMR: Generating Real-time Textual and Visual eXplanations to Facilitate UltraSonography Learning in MR](https://mlsanity.com/paper/2502.18640/) (2025)
- [Rubikon: Intelligent Tutoring for Rubik's Cube Learning Through AR-enabled Physical Task Reconfiguration](https://mlsanity.com/paper/2503.12619/) (2025)
- [Generative Lecture: Making Lecture Videos Interactive with LLMs and AI Clone Instructors](https://mlsanity.com/paper/2512.21796/) (2025)
- [WeldAR: Augmenting Live Hands-On Training with In-Situ Guidance for Novice Learners](https://mlsanity.com/paper/2603.07959/) (2026)
- [Substantial, Decomposable, and Invisible: Visual Context Misalignment in Instructional Videos for Physical Tasks](https://mlsanity.com/paper/2605.17184/) (2026)
- [RegulAR: Graph-Grounded Error Recognition and Assistance for Procedural Tasks in AR](https://mlsanity.com/paper/2608.26715/) (2026)

## Abstract (arXiv)

Visual instructions for physical tasks are typically authored in one context and followed in another, requiring users to translate demonstrated tools, materials, and spatial relationships into their own environment. We introduce Generative Tutorial, a conceptual framework for live visual instruction that depicts intended outcomes and actions within the user's environment and task flow. A formative evaluation of state-of-the-art image and video generation identifies failures and potential benefits across 15 physical tasks. Drawing on these findings, we build an augmented-reality prototype system that proactively generates goal images and demonstration videos using observed workspace context and predicted visual outcomes of preceding actions. A 24-participant lab study found higher task performance quality, greater perceived workspace correspondence, and shorter step-confirmation intervals with the system than with pre-authored guidance. Qualitative findings highlighted how contextual resemblance shapes trust, how generation errors affect interpretation, and how guidance delivery should adapt to users' needs, informing future designs.