mlsanityData to 7 Oct 2026

SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations

arXiv 2104.08667 ↗First posted 18 Apr 2021cs.CL cs.AIcited by 54

Not on the map. Why?

Abstract (arXiv)

Next generation task-oriented dialog systems need to understand conversational contexts with their perceived surroundings, to effectively help users in the real-world multimodal environment. Existing task-oriented dialog datasets aimed towards virtual assistance fall short and do not situate the dialog in the user's multimodal context. To overcome, we present a new dataset for Situated and Interactive Multimodal Conversations, SIMMC 2.0, which includes 11K task-oriented user<->assistant dialogs (117K utterances) in the shopping domain, grounded in immersive and photo-realistic scenes. The dialogs are collected using a two-phase pipeline: (1) A novel multimodal dialog simulator generates simulated dialog flows, with an emphasis on diversity and richness of interactions, (2) Manual paraphrasing of the generated utterances to collect diverse referring expressions. We provide an in-depth analysis of the collected dataset, and describe in detail the four main benchmark tasks we propose. Our baseline model, powered by the state-of-the-art language model, shows promising results, and highlights new challenges and directions for the community to study.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 16 of its 35 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

16 of 35 references.

References · 16

  1. 2015VQA: Visual Question Answeringcited by 2,489
  2. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  3. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  4. 2016Visual Dialogcited by 377
  5. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  6. 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 112
  7. 2020SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teachingcited by 71
  8. 2018Building a Conversational Agent Overnight with Dialogue Self-Playcited by 70
  9. 2019Dialog State Tracking: A Neural Reading Comprehension Approachcited by 62
  10. 2019Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systemscited by 52
  11. 2020Overview of the Ninth Dialog System Technology Challenge: DSTC9cited by 49
  12. 2018Talk the Walk: Navigating New York City through Grounded Dialoguecited by 44
4 more
  1. 2019CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialogcited by 43
  2. 2019BERT-DST: Scalable End-to-End Dialogue State Tracking with Bidirectional Encoder Representations from Transformercited by 37
  3. 2020Situated and Interactive Multimodal Conversationscited by 35
  4. 2020A Response Retrieval Approach for Dialogue Using a Multi-Attentive Transformercited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 12; 2023: 10; 2024: 10; 2025: 8; 2026: 7.

54 papers.

Sort:

By area:

  1. 2021Few-Shot Bot: Prompt-Based Learning for Dialogue Systemscited by 64
  2. 2021BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modelingcited by 55
  3. 2022SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generationcited by 34
  4. 2022SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understandingcited by 28
  5. 2024MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversationscited by 25
  6. 2022"Do you follow me?": A Survey of Recent Approaches in Dialogue State Trackingcited by 22
  7. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 22
  8. 2023PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Expertscited by 19
  9. 2023End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directionscited by 17
  10. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  11. 2022Dialog Acts for Task-Driven Embodied Agentscited by 13
  12. 2021Database Search Results Disambiguation for Task-Oriented Dialog Systemscited by 9
38 more
  1. 2022CGoDial: A Large-Scale Benchmark for Chinese Goal-oriented Dialog Evaluationcited by 9
  2. 2025AmbiK: Dataset of Ambiguous Tasks in Kitchen Environmentcited by 8
  3. 2024Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!cited by 7
  4. 2022Multimodal Dialogue State Trackingcited by 6
  5. 2023SPRING: Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graphcited by 6
  6. 2024J-CRe3: A Japanese Conversation Dataset for Real-world Reference Resolutioncited by 6
  7. 2021The JDDC 2.0 Corpus: A Large-Scale Multimodal Multi-Turn Chinese Dialogue Dataset for E-commerce Customer Servicecited by 5
  8. 2024Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requestscited by 4
  9. 2024TOAD: Task-Oriented Automatic Dialogs with Diverse Response Stylescited by 4
  10. 2026Evaluating Scene-based In-Situ Item Labeling for Immersive Conversational Recommendationcited by 4
  11. 2023'What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchangescited by 3
  12. 2023UniPCM: Universal Pre-trained Conversation Model with Task-aware Automatic Promptcited by 3
  13. 2024OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialogcited by 3
  14. 2024Multi-Modal Video Dialog State Tracking in the Wildcited by 3
  15. 2024Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Modelscited by 3
  16. 2025Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactionscited by 3
  17. 2021UNITER-Based Situated Coreference Resolution with Rich Multimodal Inputcited by 2
  18. 2022Exploring Multi-Modal Representations for Ambiguity Detection & Coreference Resolution in the SIMMC 2.0 Challengecited by 2
  19. 2021Multimodal Interactions Using Pretrained Unimodal Models for SIMMC 2.0cited by 1
  20. 2022Tell Your Story: Task-Oriented Dialogs for Interactive Content Creationcited by 1
  21. 2023Which One Are You Referring To? Multimodal Object Identification in Situated Dialoguecited by 1
  22. 2023SimpleMTOD: A Simple Language Model for Multimodal Task-Oriented Dialogue with Symbolic Scene Representationcited by 1
  23. 2023Collecting Visually-Grounded Dialogue with A Game Of Sortscited by 1
  24. 2024Task Oriented Dialogue as a Catalyst for Self-Supervised Automatic Speech Recognitioncited by 1
  25. 2025Multimodal Coreference Resolution for Chinese Social Media Dialogues: Dataset and Benchmark Approachcited by 1
  26. 2025PicPersona-TOD : A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Personacited by 1
  27. 2025Beyond Words: Infusing Conversational Agents with Human-like Typing Behaviorscited by 1
  28. 2026Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluationcited by 1
  29. 2026Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interactioncited by 1
  30. 2021Building Goal-Oriented Dialogue Systems with Situated Visual Contextcited by 0
  31. 2022Spot the Difference: A Cooperative Object-Referring Game in Non-Perfectly Co-Observable Scenecited by 0
  32. 2022GRILLBot: An Assistant for Real-World Tasks with Neural Semantic Parsing and Graph-Based Representationscited by 0
  33. 2022Navigating Connected Memories with a Task-oriented Dialog Systemcited by 0
  34. 2023Multimodal Recommendation Dialog with Subjective Preference: A New Challenge and Benchmarkcited by 0
  35. 2023VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialoguecited by 0
  36. 2024Multimodal Contextualized Semantic Parsing from Speechcited by 0
  37. 2025Disambiguating Reference in Visually Grounded Dialogues through Joint Modeling of Textual and Multimodal Semantic Structurescited by 0
  38. 2025J-ORA: A Framework and Multimodal Dataset for Japanese Object Identification, Reference, Action Prediction in Robot Perceptioncited by 0

50 most cited of 54 All 54 (JSON)