# An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games

arXiv: https://arxiv.org/abs/2102.00424 · HTML: https://mlsanity.com/paper/2102.00424/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 31 Jan 2021.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [12-in-1: Multi-Task Vision and Language Representation Learning](https://mlsanity.com/paper/1912.02315/) (2019)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [UNITER: UNiversal Image-TExt Representation Learning](https://mlsanity.com/paper/1909.11740/) (2019)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [An Analysis of Visual Question Answering Algorithms](https://mlsanity.com/paper/1703.09684/) (2017)
- [MUREL: Multimodal Relational Reasoning for Visual Question Answering](https://mlsanity.com/paper/1902.09487/) (2019)
- [Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries](https://mlsanity.com/paper/1711.06370/) (2017)
- [Countering Language Drift via Visual Grounding](https://mlsanity.com/paper/1909.04499/) (2019)
- [A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks](https://mlsanity.com/paper/1811.06031/) (2018)
- [End-to-end optimization of goal-driven and visually grounded dialogue systems](https://mlsanity.com/paper/1703.05423/) (2017)
- [Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat](https://mlsanity.com/paper/1809.03408/) (2018)
- [Training Recurrent Answering Units with Joint Loss Minimization for VQA](https://mlsanity.com/paper/1606.03647/) (2016)
- [CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning](https://mlsanity.com/paper/2006.02174/) (2020)
- [Dialog without Dialog Data: Learning Visual Dialog Agents from VQA Data](https://mlsanity.com/paper/2007.12750/) (2020)
- [Ask No More: Deciding when to guess in referential visual dialogue](https://mlsanity.com/paper/1805.06960/) (2018)

## Built on by (most cited first, 1 of 1)

- ['What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchanges](https://mlsanity.com/paper/2307.15554/) (2023)

## Abstract (arXiv)

Guessing games are a prototypical instance of the "learning by interacting" paradigm. This work investigates how well an artificial agent can benefit from playing guessing games when later asked to perform on novel NLP downstream tasks such as Visual Question Answering (VQA). We propose two ways to exploit playing guessing games: 1) a supervised learning scenario in which the agent learns to mimic successful guessing games and 2) a novel way for an agent to play by itself, called Self-play via Iterated Experience Learning (SPIEL). We evaluate the ability of both procedures to generalize: an in-domain evaluation shows an increased accuracy (+7.79) compared with competitors on the evaluation suite CompGuessWhat?!; a transfer evaluation shows improved performance for VQA on the TDIUC dataset in terms of harmonic average accuracy (+5.31) thanks to more fine-grained object representations learned via SPIEL.