# The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents

arXiv: https://arxiv.org/abs/1911.03768 · HTML: https://mlsanity.com/paper/1911.03768/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2019.

## Built on by (most cited first, 30 of 45)

- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [Hash Layers For Large Sparse Models](https://mlsanity.com/paper/2106.04426/) (2021)
- [BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage](https://mlsanity.com/paper/2208.03188/) (2022)
- [Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering](https://mlsanity.com/paper/2104.08202/) (2021)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion](https://mlsanity.com/paper/2203.13224/) (2022)
- [Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features](https://mlsanity.com/paper/2107.06963/) (2021)
- [Few-Shot Bot: Prompt-Based Learning for Dialogue Systems](https://mlsanity.com/paper/2110.08118/) (2021)
- [Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes](https://mlsanity.com/paper/2109.08828/) (2021)
- [Prompt Stealing Attacks Against Large Language Models](https://mlsanity.com/paper/2402.12959/) (2024)
- [Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attacks](https://mlsanity.com/paper/2307.16630/) (2023)
- [Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots](https://mlsanity.com/paper/2209.03463/) (2022)
- [MMChat: Multi-Modal Chat Dataset on Social Media](https://mlsanity.com/paper/2108.07154/) (2021)
- [Call for Customized Conversation: Customized Conversation Grounding Persona and Knowledge](https://mlsanity.com/paper/2112.08619/) (2021)
- [Fusing task-oriented and open-domain dialogues in conversational agents](https://mlsanity.com/paper/2109.04137/) (2021)
- [Learning to Memorize Entailment and Discourse Relations for Persona-Consistent Dialogues](https://mlsanity.com/paper/2301.04871/) (2023)
- [Staircase Attention for Recurrent Processing of Sequences](https://mlsanity.com/paper/2106.04279/) (2021)
- [BotsTalk: Machine-sourced Framework for Automatic Curation of Large-scale Multi-skill Dialogue Datasets](https://mlsanity.com/paper/2210.12687/) (2022)
- [MidMed: Towards Mixed-Type Dialogues for Medical Consultation](https://mlsanity.com/paper/2306.02923/) (2023)
- [Answer-Me: Multi-Task Open-Vocabulary Visual Question Answering](https://mlsanity.com/paper/2205.00949/) (2022)
- [FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue](https://mlsanity.com/paper/2205.06262/) (2022)
- [Going for GOAL: A Resource for Grounded Football Commentaries](https://mlsanity.com/paper/2211.04534/) (2022)
- [The Web Can Be Your Oyster for Improving Large Language Models](https://mlsanity.com/paper/2305.10998/) (2023)
- [WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning](https://mlsanity.com/paper/2212.10057/) (2022)
- [PK-ICR: Persona-Knowledge Interactive Context Retrieval for Grounded Dialogue](https://mlsanity.com/paper/2302.06674/) (2023)
- [SideControl: Controlled Open-domain Dialogue Generation via Additive Side Networks](https://mlsanity.com/paper/2109.01958/) (2021)
- [FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows](https://mlsanity.com/paper/2202.06633/) (2022)
- [A Critical Reflection and Forward Perspective on Empathy and Natural Language Processing](https://mlsanity.com/paper/2210.16604/) (2022)

## Abstract (arXiv)

We introduce dodecaDialogue: a set of 12 tasks that measures if a conversational agent can communicate engagingly with personality and empathy, ask questions, answer questions by utilizing knowledge resources, discuss topics and situations, and perceive and converse about images. By multi-tasking on such a broad large-scale set of data, we hope to both move towards and measure progress in producing a single unified agent that can perceive, reason and converse with humans in an open-domain setting. We show that such multi-tasking improves over a BERT pre-trained baseline, largely due to multi-tasking with very large dialogue datasets in a similar domain, and that the multi-tasking in general provides gains to both text and image-based tasks using several metrics in both the fine-tune and task transfer settings. We obtain state-of-the-art results on many of the tasks, providing a strong baseline for this challenge.