# The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues

arXiv: https://arxiv.org/abs/2103.11151 · HTML: https://mlsanity.com/paper/2103.11151/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 20 Mar 2021.

## Builds on (most cited first, 30 of 31)

- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://mlsanity.com/paper/1706.05125/) (2017)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [What makes a good conversation? How controllable attributes affect human judgments](https://mlsanity.com/paper/1902.08654/) (2019)
- [Unifying Human and Statistical Evaluation for Natural Language Generation](https://mlsanity.com/paper/1904.02792/) (2019)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training](https://mlsanity.com/paper/1911.03860/) (2019)
- [Adversarial Learning for Neural Dialogue Generation](https://mlsanity.com/paper/1701.06547/) (2017)
- [Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses](https://mlsanity.com/paper/1708.07149/) (2017)
- [Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.06585/) (2017)
- [Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation](https://mlsanity.com/paper/1701.08251/) (2017)
- [Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings](https://mlsanity.com/paper/1704.07130/) (2017)
- [The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue](https://mlsanity.com/paper/1906.01530/) (2019)
- [Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog](https://mlsanity.com/paper/1902.00579/) (2019)
- [Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation](https://mlsanity.com/paper/1909.04696/) (2019)
- [A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context](https://mlsanity.com/paper/1907.03399/) (2019)
- [Improving Generative Visual Dialog by Answering Diverse Questions](https://mlsanity.com/paper/1909.10470/) (2019)
- [End-to-end optimization of goal-driven and visually grounded dialogue systems](https://mlsanity.com/paper/1703.05423/) (2017)
- [Topic-based Evaluation for Conversational Bots](https://mlsanity.com/paper/1801.03622/) (2018)
- [Self-Supervised Dialogue Learning](https://mlsanity.com/paper/1907.00448/) (2019)
- [Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat](https://mlsanity.com/paper/1809.03408/) (2018)
- [What Should I Ask? Using Conversationally Informative Rewards for Goal-Oriented Visual Dialog](https://mlsanity.com/paper/1907.12021/) (2019)
- [Visual Dialogue State Tracking for Question Generation](https://mlsanity.com/paper/1911.07928/) (2019)
- [Adversarial Evaluation of Dialogue Models](https://mlsanity.com/paper/1701.08198/) (2017)
- [Multimodal Hierarchical Reinforcement Learning Policy for Task-Oriented Visual Dialog](https://mlsanity.com/paper/1805.03257/) (2018)
- [Building Task-Oriented Visual Dialog Systems Through Alternative Optimization Between Dialog Policy and Language Generation](https://mlsanity.com/paper/1909.05365/) (2019)
- [What's to know? Uncertainty as a Guide to Asking Goal-oriented Questions](https://mlsanity.com/paper/1812.06401/) (2018)

## Built on by (most cited first, 2 of 2)

- [Looking for Confirmations: An Effective and Human-Like Visual Dialogue Strategy](https://mlsanity.com/paper/2109.05312/) (2021)
- [Are Current Decoding Strategies Capable of Facing the Challenges of Visual Dialogue?](https://mlsanity.com/paper/2210.12997/) (2022)

## Abstract (arXiv)

When training a model on referential dialogue guessing games, the best model is usually chosen based on its task success. We show that in the popular end-to-end approach, this choice prevents the model from learning to generate linguistically richer dialogues, since the acquisition of language proficiency takes longer than learning the guessing task. By comparing models playing different games (GuessWhat, GuessWhich, and Mutual Friends), we show that this discrepancy is model- and task-agnostic. We investigate whether and when better language quality could lead to higher task success. We show that in GuessWhat, models could increase their accuracy if they learn to ground, encode, and decode also words that do not occur frequently in the training set.