# Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?

arXiv: https://arxiv.org/abs/2311.00738 · HTML: https://mlsanity.com/paper/2311.00738/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [Simple but Effective: CLIP Embeddings for Embodied AI](https://mlsanity.com/paper/2111.09888/) (2021)
- [MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound](https://mlsanity.com/paper/2201.02639/) (2022)
- [Vision-Language Models as Success Detectors](https://mlsanity.com/paper/2303.07280/) (2023)
- [Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos](https://mlsanity.com/paper/1804.09626/) (2018)
- [DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics](https://mlsanity.com/paper/2210.02438/) (2022)
- [MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks](https://mlsanity.com/paper/2109.06275/) (2021)
- [Situated and Interactive Multimodal Conversations](https://mlsanity.com/paper/2006.01460/) (2020)
- [Detecting Engagement in Egocentric Video](https://mlsanity.com/paper/1604.00906/) (2016)
- [Automated capture and delivery of assistive task guidance with an eyewear computer: The GlaciAR system](https://mlsanity.com/paper/1701.02586/) (2016)
- [CAISE: Conversational Agent for Image Search and Editing](https://mlsanity.com/paper/2202.11847/) (2022)

## Built on by (most cited first, 18 of 18)

- [Proactive Assistant Dialogue Generation from Streaming Egocentric Videos](https://mlsanity.com/paper/2506.05904/) (2025)
- [Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties](https://mlsanity.com/paper/2311.17041/) (2023)
- [What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction](https://mlsanity.com/paper/2407.08101/) (2024)
- [SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Research](https://mlsanity.com/paper/2405.13035/) (2024)
- [Can Vision-Language Models Answer Face to Face Questions in the Real-World?](https://mlsanity.com/paper/2503.19356/) (2025)
- [HANDI: Hand-Centric Text-and-Image Conditioned Video Generation](https://mlsanity.com/paper/2412.04189/) (2024)
- [Transparent and Coherent Procedural Mistake Detection](https://mlsanity.com/paper/2412.11927/) (2024)
- [Towards Effective Human-in-the-Loop Assistive AI Agents](https://mlsanity.com/paper/2507.18374/) (2025)
- [Harnessing Streaming Video in the Wild](https://mlsanity.com/paper/2606.08615/) (2026)
- [Pro2Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural Tasks](https://mlsanity.com/paper/2605.04227/) (2026)
- ["Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaboration](https://mlsanity.com/paper/2409.10525/) (2024)
- [Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?](https://mlsanity.com/paper/2511.21998/) (2025)
- [POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models](https://mlsanity.com/paper/2406.03843/) (2024)
- [Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU](https://mlsanity.com/paper/2602.15707/) (2026)
- [Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?](https://mlsanity.com/paper/2606.09547/) (2026)
- [GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video](https://mlsanity.com/paper/2607.02991/) (2026)
- [Temporally-Grounded Language Generation: Towards Real-Time Vision-Language Models](https://mlsanity.com/paper/2505.11326/) (2025)
- [SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration](https://mlsanity.com/paper/2511.02560/) (2025)

## Abstract (arXiv)

Despite tremendous advances in AI, it remains a significant challenge to develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in various tasks. These systems need to have a sophisticated understanding of the user as well as the environment, and make timely accurate decisions on when and what to say. To address this issue, we created a new multimodal benchmark dataset, Watch, Talk and Guide (WTaG) based on natural interaction between a human user and a human instructor. We further proposed two tasks: User and Environment Understanding, and Instructor Decision Making. We leveraged several foundation models to study to what extent these models can be quickly adapted to perceptually enabled task guidance. Our quantitative, qualitative, and human evaluation results show that these models can demonstrate fair performances in some cases with no task-specific training, but a fast and reliable adaptation remains a significant challenge. Our benchmark and baselines will provide a stepping stone for future work on situated task guidance.