# Situated and Interactive Multimodal Conversations

arXiv: https://arxiv.org/abs/2006.01460 · HTML: https://mlsanity.com/paper/2006.01460/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2020.

## Built on by (most cited first, 30 of 35)

- [SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations](https://mlsanity.com/paper/2104.08667/) (2021)
- [BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling](https://mlsanity.com/paper/2106.02787/) (2021)
- [Zero-Shot Dialogue State Tracking via Cross-Task Transfer](https://mlsanity.com/paper/2109.04655/) (2021)
- [Proactive Assistant Dialogue Generation from Streaming Egocentric Videos](https://mlsanity.com/paper/2506.05904/) (2025)
- [Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?](https://mlsanity.com/paper/2311.00738/) (2023)
- [Multimodal Conversational AI: A Survey of Datasets and Approaches](https://mlsanity.com/paper/2205.06907/) (2022)
- [DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue](https://mlsanity.com/paper/2101.00151/) (2021)
- [cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers](https://mlsanity.com/paper/2406.08398/) (2024)
- [Database Search Results Disambiguation for Task-Oriented Dialog Systems](https://mlsanity.com/paper/2112.08351/) (2021)
- [Maintaining Common Ground in Dynamic Environments](https://mlsanity.com/paper/2105.14207/) (2021)
- [Going for GOAL: A Resource for Grounded Football Commentaries](https://mlsanity.com/paper/2211.04534/) (2022)
- [SPRING: Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graph](https://mlsanity.com/paper/2301.01949/) (2023)
- [Decision-Theoretic Question Generation for Situated Reference Resolution: An Empirical Study and Computational Model](https://mlsanity.com/paper/2110.06288/) (2021)
- [CAISE: Conversational Agent for Image Search and Editing](https://mlsanity.com/paper/2202.11847/) (2022)
- [OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog](https://mlsanity.com/paper/2402.13146/) (2024)
- [Multi-Modal Video Dialog State Tracking in the Wild](https://mlsanity.com/paper/2407.02218/) (2024)
- [Simple Dialogue System with AUDITED](https://mlsanity.com/paper/2110.11881/) (2021)
- [I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialogue](https://mlsanity.com/paper/2503.00071/) (2025)
- [Multi-Task Learning for Situated Multi-Domain End-to-End Dialogue Systems](https://mlsanity.com/paper/2110.05221/) (2021)
- [Multimodal Interactions Using Pretrained Unimodal Models for SIMMC 2.0](https://mlsanity.com/paper/2112.05328/) (2021)
- [Which One Are You Referring To? Multimodal Object Identification in Situated Dialogue](https://mlsanity.com/paper/2302.14680/) (2023)
- [A Unified Framework for Slot based Response Generation in a Multimodal Dialogue System](https://mlsanity.com/paper/2305.17433/) (2023)
- [SimpleMTOD: A Simple Language Model for Multimodal Task-Oriented Dialogue with Symbolic Scene Representation](https://mlsanity.com/paper/2307.04907/) (2023)
- [Learning Communication Policies for Different Follower Behaviors in a Collaborative Reference Game](https://mlsanity.com/paper/2402.04824/) (2024)
- [PSCon: Product Search Through Conversations](https://mlsanity.com/paper/2502.13881/) (2025)
- [Building Goal-Oriented Dialogue Systems with Situated Visual Context](https://mlsanity.com/paper/2111.11576/) (2021)
- [Spot the Difference: A Cooperative Object-Referring Game in Non-Perfectly Co-Observable Scene](https://mlsanity.com/paper/2203.08362/) (2022)
- [Navigating Connected Memories with a Task-oriented Dialog System](https://mlsanity.com/paper/2211.08462/) (2022)
- [From Knowledge Augmentation to Multi-tasking: Towards Human-like Dialogue Systems](https://mlsanity.com/paper/2212.03279/) (2022)
- [Multimodal Recommendation Dialog with Subjective Preference: A New Challenge and Benchmark](https://mlsanity.com/paper/2305.18212/) (2023)

## Abstract (arXiv)

Next generation virtual assistants are envisioned to handle multimodal inputs (e.g., vision, memories of previous interactions, in addition to the user's utterances), and perform multimodal actions (e.g., displaying a route in addition to generating the system's utterance). We introduce Situated Interactive MultiModal Conversations (SIMMC) as a new direction aimed at training agents that take multimodal actions grounded in a co-evolving multimodal input context in addition to the dialog history. We provide two SIMMC datasets totalling ~13K human-human dialogs (~169K utterances) using a multimodal Wizard-of-Oz (WoZ) setup, on two shopping domains: (a) furniture (grounded in a shared virtual environment) and, (b) fashion (grounded in an evolving set of images). We also provide logs of the items appearing in each scene, and contextual NLU and coreference annotations, using a novel and unified framework of SIMMC conversational acts for both user and assistant utterances. Finally, we present several tasks within SIMMC as objective evaluation protocols, such as Structural API Prediction and Response Generation. We benchmark a collection of existing models on these SIMMC tasks as strong baselines, and demonstrate rich multimodal conversational interactions. Our data, annotations, code, and models are publicly available.