mlsanityData to 14 Oct 2026

Multimodal Conversational AI: A Survey of Datasets and Approaches

arXiv 2205.06907 ↗First posted 13 May 2022cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

As humans, we experience the world with all our senses or modalities (sound, sight, touch, smell, and taste). We use these modalities, particularly sight and touch, to convey and interpret specific meanings. Multimodal expressions are central to conversations; a rich set of modalities amplify and often compensate for each other. A multimodal conversational AI system answers questions, fulfills tasks, and emulates human conversations by understanding and expressing itself via multiple modalities. This paper motivates, defines, and mathematically formulates the multimodal conversational research objective. We provide a taxonomy of research required to solve the objective: multimodal representation, fusion, alignment, translation, and co-learning. We survey state-of-the-art datasets and approaches for each research area and highlight their limiting assumptions. Finally, we identify multimodal co-learning as a promising direction for multimodal conversational AI research.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 101 of its 173 references that are in the map. First posted on arXiv 13 May 2022.

Builds on

101 of 173 references.

References · 101

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  12. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
38 more
  1. 2015VQA: Visual Question Answeringcited by 2,506
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  8. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  10. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  11. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  13. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  14. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  15. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  16. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  17. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  18. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
  19. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  20. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  21. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  22. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  23. 2020Multi-modal Transformer for Video Retrievalcited by 395
  24. 2016Visual Dialogcited by 377
  25. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  26. 2020A Simple Language Model for Task-Oriented Dialoguecited by 332
  27. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  28. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 310
  29. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  30. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  31. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  32. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
  33. 2019Vision-and-Dialog Navigationcited by 250
  34. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  35. 2016A Persona-Based Neural Conversation Modelcited by 223
  36. 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
  37. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
  38. 2015A Neural Conversational Modelcited by 196

50 most cited of 101 All 101 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 8; 2024: 4; 2025: 1; 2026: 1.

16 papers.

Sort:

By area:

  1. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  2. 2023FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Modelscited by 46
  3. 2022"Do you follow me?": A Survey of Recent Approaches in Dialogue State Trackingcited by 22
  4. 2023Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategycited by 12
  5. 2024cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Paperscited by 11
  6. 2022Interactive Question Answering Systems: Literature Reviewcited by 7
  7. 2023cTBLS: Augmenting Large Language Models with Conversational Tablescited by 7
  8. 2023Bringing the State-of-the-Art to Customers: A Neural Agent Assistant Framework for Customer Service Supportcited by 6
  9. 2024iTBLS: A Dataset of Interactive Conversations Over Tabular Informationcited by 3
  10. 2024OpenOmni: A Collaborative Open Source Tool for Building Future-Ready Multimodal Conversational Agentscited by 3
  11. 2024Attend-Fusion: Efficient Audio-Visual Fusion for Video Classificationcited by 3
  12. 2025Cross-Format Retrieval-Augmented Generation in XR with LLMs for Context-Aware Maintenance Assistancecited by 3
4 more
  1. 2023Which One Are You Referring To? Multimodal Object Identification in Situated Dialoguecited by 1
  2. 2023Who's in Charge? Roles and Responsibilities of Decision-Making Components in Conversational Robotscited by 1
  3. 2023A Contextualized Real-Time Multimodal Emotion Recognition for Conversational Agents using Graph Convolutional Networks in Reinforcement Learningcited by 1
  4. 2026POPS: Recovering Unlearned Multi-Modality Knowledge in MLLMs with Prompt-Optimized Parameter Shakingcited by 0