# All-in-One Image-Grounded Conversational Agents

arXiv: https://arxiv.org/abs/1912.12394 · HTML: https://mlsanity.com/paper/1912.12394/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2019.

## Built on by (most cited first, 3 of 3)

- [MMChat: Multi-Modal Chat Dataset on Social Media](https://mlsanity.com/paper/2108.07154/) (2021)
- [i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data](https://mlsanity.com/paper/2305.12311/) (2023)
- [A Unified Framework for Slot based Response Generation in a Multimodal Dialogue System](https://mlsanity.com/paper/2305.17433/) (2023)

## Abstract (arXiv)

As single-task accuracy on individual language and image tasks has improved substantially in the last few years, the long-term goal of a generally skilled agent that can both see and talk becomes more feasible to explore. In this work, we focus on leveraging individual language and image tasks, along with resources that incorporate both vision and language towards that objective. We design an architecture that combines state-of-the-art Transformer and ResNeXt modules fed into a novel attentive multimodal module to produce a combined model trained on many tasks. We provide a thorough analysis of the components of the model, and transfer performance when training on one, some, or all of the tasks. Our final models provide a single system that obtains good results on all vision and language tasks considered, and improves the state-of-the-art in image-grounded conversational applications.