# Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz

arXiv: https://arxiv.org/abs/2510.06872 · HTML: https://mlsanity.com/paper/2510.06872/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 64 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models](https://mlsanity.com/paper/2402.14848/) (2024)
- [ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing](https://mlsanity.com/paper/2309.09128/) (2023)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience](https://mlsanity.com/paper/2302.10395/) (2023)
- [Next Steps for Human-Centered Generative AI: A Technical Perspective](https://mlsanity.com/paper/2306.15774/) (2023)
- [Probabilistic Soundness Guarantees in LLM Reasoning Chains](https://mlsanity.com/paper/2507.12948/) (2025)
- [The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users](https://mlsanity.com/paper/1301.7385/) (2013)
- [Jigsaw: Supporting Designers to Prototype Multimodal Applications by Chaining AI Foundation Models](https://mlsanity.com/paper/2310.08574/) (2023)
- [Exploring Prompt Engineering Practices in the Enterprise](https://mlsanity.com/paper/2403.08950/) (2024)
- [Exploring the Potential of Metacognitive Support Agents for Human-AI Co-Creation](https://mlsanity.com/paper/2506.12879/) (2025)
- [Addressing UX Practitioners' Challenges in Designing ML Applications: an Interactive Machine Learning Approach](https://mlsanity.com/paper/2302.11843/) (2023)
- [In Situ AI Prototyping: Infusing Multimodal Prompts into Mobile Settings with MobileMaker](https://mlsanity.com/paper/2405.03806/) (2024)

## Built on by (most cited first, 2 of 2)

- [DigitalCoach: Communication and Grounding Gaps in Human and Agentic Computer Use Coaching](https://mlsanity.com/paper/2606.31980/) (2026)
- [Generative Tutorial: Towards Live Contextualized Visual Instructions for Physical Tasks](https://mlsanity.com/paper/2609.24955/) (2026)

## Abstract (arXiv)

Recent advancements in multimodal generative AI (GenAI) enable the creation of personal context-aware real-time agents that, for example, can augment user workflows by following their on-screen activities and providing contextual assistance. However, prototyping such experiences is challenging, especially when supporting people with domain-specific tasks using real-time inputs such as speech and screen recordings. While prototyping an LLM-based proactive support agent system, we found that existing prototyping and evaluation methods were insufficient to anticipate the nuanced situational complexity and contextual immediacy required. To overcome these challenges, we explored a novel user-centered prototyping approach that combines counterfactual video replay prompting and hybrid Wizard-of-Oz methods to iteratively design and refine agent behaviors. This paper discusses our prototyping experiences, highlighting successes and limitations, and offers a practical guide and an open-source toolkit for UX designers, HCI researchers, and AI toolmakers to build more user-centered and context-aware multimodal agents.