# Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1703.06585 · HTML: https://mlsanity.com/paper/1703.06585/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2017.

## Built on by (most cited first, 30 of 71)

- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [Offline RL for Natural Language Generation with Implicit Language Q Learning](https://mlsanity.com/paper/2206.11871/) (2022)
- [Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.00783/) (2021)
- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistant](https://mlsanity.com/paper/2203.04203/) (2022)
- [Survey on reinforcement learning for language processing](https://mlsanity.com/paper/2104.05565/) (2021)
- [Interactive Learning from Activity Description](https://mlsanity.com/paper/2102.07024/) (2021)
- [Chatting Makes Perfect: Chat-based Image Retrieval](https://mlsanity.com/paper/2305.20062/) (2023)
- [ChatShop: Interactive Information Seeking with Language Agents](https://mlsanity.com/paper/2404.09911/) (2024)
- [Learning to Draw: Emergent Communication through Sketching](https://mlsanity.com/paper/2106.02067/) (2021)
- [GridToPix: Training Embodied Agents with Minimal Supervision](https://mlsanity.com/paper/2105.00931/) (2021)
- [Open-domain clarification question generation without question examples](https://mlsanity.com/paper/2110.09779/) (2021)
- [Maria: A Visual Experience Powered Conversational Agent](https://mlsanity.com/paper/2105.13073/) (2021)
- [Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents](https://mlsanity.com/paper/2110.05769/) (2021)
- [Learning to Retrieve Videos by Asking Questions](https://mlsanity.com/paper/2205.05739/) (2022)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts](https://mlsanity.com/paper/2109.12761/) (2021)
- [SEDM: Selective Self-Evolving Distributed Memory for Agents](https://mlsanity.com/paper/2509.09498/) (2025)
- [Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learning](https://mlsanity.com/paper/2110.06894/) (2021)
- [Towards More Human-like AI Communication: A Review of Emergent Communication Research](https://mlsanity.com/paper/2308.02541/) (2023)
- [Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication](https://mlsanity.com/paper/2111.06464/) (2021)
- [Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networks](https://mlsanity.com/paper/2206.02887/) (2022)
- [Communication Drives the Emergence of Language Universals in Neural Agents: Evidence from the Word-order/Case-marking Trade-off](https://mlsanity.com/paper/2301.13083/) (2023)
- [Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategy](https://mlsanity.com/paper/2304.07007/) (2023)
- [Learning to Generate Equitable Text in Dialogue from Biased Training Data](https://mlsanity.com/paper/2307.04303/) (2023)
- [Simple Baselines for Interactive Video Retrieval with Questions and Answers](https://mlsanity.com/paper/2308.10402/) (2023)
- [ImageScope: Unifying Language-Guided Image Retrieval via Large Multimodal Model Collective Reasoning](https://mlsanity.com/paper/2503.10166/) (2025)
- [Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook](https://mlsanity.com/paper/2210.13623/) (2022)
- [Ask&Confirm: Active Detail Enriching for Cross-Modal Retrieval with Partial Query](https://mlsanity.com/paper/2103.01654/) (2021)
- [Controllable Video Captioning with an Exemplar Sentence](https://mlsanity.com/paper/2112.01073/) (2021)

## Abstract (arXiv)

We introduce the first goal-driven training for visual question answering and dialog agents. Specifically, we pose a cooperative 'image guessing' game between two agents -- Qbot and Abot -- who communicate in natural language dialog so that Qbot can select an unseen image from a lineup of images. We use deep reinforcement learning (RL) to learn the policies of these agents end-to-end -- from pixels to multi-agent multi-round dialog to game reward. We demonstrate two experimental results. First, as a 'sanity check' demonstration of pure RL (from scratch), we show results on a synthetic world, where the agents communicate in ungrounded vocabulary, i.e., symbols with no pre-specified meanings (X, Y, Z). We find that two bots invent their own communication protocol and start using certain symbols to ask/answer about certain visual attributes (shape/color/style). Thus, we demonstrate the emergence of grounded language and communication among 'visual' dialog agents with no human supervision. Second, we conduct large-scale real-image experiments on the VisDial dataset, where we pretrain with supervised dialog data and show that the RL 'fine-tuned' agents significantly outperform SL agents. Interestingly, the RL Qbot learns to ask questions that Abot is good at, ultimately resulting in more informative dialog and a better team.