mlsanityData to 7 Oct 2026

Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation

arXiv 2105.14445 ↗First posted 30 May 2021cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Multi-modal dialog modeling is of growing interest. In this work, we propose frameworks to resolve a specific case of multi-modal dialog generation that better mimics multi-modal dialog generation in the real world, where each dialog turn is associated with the visual context in which it takes place. Specifically, we propose to model the mutual dependency between text-visual features, where the model not only needs to learn the probability of generating the next dialog utterance given preceding dialog utterances and visual contexts, but also the probability of predicting the visual features in which a dialog utterance takes place, leading the generated dialog utterance specific to the visual context. We observe significant performance boosts over vanilla models when the mutual dependency between text and visual features is modeled. Code is available at https://github.com/ShannonAI/OpenViDial.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 56 of its 79 references that are in the map. First posted on arXiv 30 May 2021.

Builds on

56 of 79 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
38 more
  1. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  2. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  3. 2020Recipes for building an open-domain chatbotcited by 593
  4. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  5. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  6. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  7. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  8. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  9. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  10. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  11. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  12. 2016A Persona-Based Neural Conversation Modelcited by 221
  13. 2019Challenges in Building Intelligent Open-domain Dialog Systemscited by 147
  14. 2016Neural Belief Tracker: Data-Driven Dialogue State Trackingcited by 142
  15. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  16. 2017Adversarial Learning for Neural Dialogue Generationcited by 103
  17. 2015A Neural Network Approach to Context-Sensitive Generation of Conversational Responsescited by 98
  18. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  19. 2021M6: A Chinese Multimodal Pretrainercited by 91
  20. 2018Image Chat: Engaging Grounded Conversationscited by 76
  21. 2019Fusion of Detected Objects in Text for Visual Question Answeringcited by 74
  22. 2017Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learningcited by 70
  23. 2018Building a Conversational Agent Overnight with Dialogue Self-Playcited by 70
  24. 2019Dialog State Tracking: A Neural Reading Comprehension Approachcited by 62
  25. 2017Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generationcited by 59
  26. 2017End-to-End Task-Completion Neural Dialogue Systemscited by 52
  27. 2020XGPT: Cross-modal Generative Pre-Training for Image Captioningcited by 51
  28. 2019Multi-step Reasoning via Recurrent Dual Attention for Visual Dialogcited by 41
  29. 2019Factor Graph Attentioncited by 41
  30. 2016Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Accesscited by 36
  31. 2019Pretraining Methods for Dialog Context Representation Learningcited by 35
  32. 2018Recursive Visual Attention in Visual Dialogcited by 34
  33. 2018Generating More Interesting Responses in Neural Conversation Models with Distributional Constraintscited by 33
  34. 2021Zero-shot Generalization in Dialog State Tracking through Generative Question Answeringcited by 33
  35. 2017Visual Reference Resolution using Attention Memory for Visual Dialogcited by 28
  36. 2020OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contextscited by 27
  37. 2019Dual Attention Networks for Visual Reference Resolution in Visual Dialogcited by 25
  38. 2019Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Modelscited by 25

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 1; 2023: 5.

8 papers.

  1. 2022DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Datasetcited by 17
  2. 2021OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contextscited by 16
  3. 2021ConRPG: Paraphrase Generation using Contexts as Regularizercited by 13
  4. 2023VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitionscited by 11
  5. 2023TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real Worldcited by 7
  6. 2023Iterative Adversarial Attack on Image-guided Story Ending Generationcited by 2
  7. 2023Syllable-level lyrics generation from melody exploiting character-level language modelcited by 2
  8. 2023Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialogcited by 1