# Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution

arXiv: https://arxiv.org/abs/2609.07594 · HTML: https://mlsanity.com/paper/2609.07594/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 7 Sep 2026.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policy](https://mlsanity.com/paper/2204.07433/) (2022)
- [End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2311.09008/) (2023)
- [Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems](https://mlsanity.com/paper/2006.01921/) (2020)
- [Human vs. Agent in Task-Oriented Conversations](https://mlsanity.com/paper/2509.17619/) (2025)
- [Task2Dial: A Novel Task and Dataset for Commonsense enhanced Task-based Dialogue Grounded in Documents](https://mlsanity.com/paper/2204.01061/) (2022)
- [GRILLBot In Practice: Lessons and Tradeoffs Deploying Large Language Models for Adaptable Conversational Task Assistants](https://mlsanity.com/paper/2402.07647/) (2024)
- [Rewriting the Script: Adapting Text Instructions for Voice Interaction](https://mlsanity.com/paper/2306.09992/) (2023)
- [Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistants](https://mlsanity.com/paper/2410.12891/) (2024)
- [TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus](https://mlsanity.com/paper/2310.02118/) (2023)
- [Leveraging Interesting Facts to Enhance User Engagement with Conversational Interfaces](https://mlsanity.com/paper/2404.06659/) (2024)
- [MMWOZ: Building Multimodal Agent for Task-oriented Dialogue](https://mlsanity.com/paper/2511.12586/) (2025)

## Abstract (arXiv)

Conversational Task Assistants (CTAs) are multimodal dialogue systems that support users in complex real-world tasks such as cooking and DIY through voice, text, image, and video interactions. Prior user studies have focused on controlled settings, leaving limited understanding of real-world CTA usage at scale. In this work, we present a large-scale study of CTA usage based on thousands of users in-the-wild. Our large-scale real-world data analysis unveils new understandings of (i) user-CTA interaction flows, (ii) user intents, (iii) user conversational traits, and (iv) behavioral factors associated with user satisfaction. Our findings reveal key opportunities for future research in CTAs, particularly in user interaction design and task engagement, concluding with concrete design guidelines.