The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues
Not on the map. Why?
Abstract (arXiv)
When training a model on referential dialogue guessing games, the best model is usually chosen based on its task success. We show that in the popular end-to-end approach, this choice prevents the model from learning to generate linguistically richer dialogues, since the acquisition of language proficiency takes longer than learning the guessing task. By comparing models playing different games (GuessWhat, GuessWhich, and Mutual Friends), we show that this discrepancy is model- and task-agnostic. We investigate whether and when better language quality could lead to higher task success. We show that in GuessWhat, models could increase their accuracy if they learn to ground, encode, and decode also words that do not occur frequently in the training set.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 38 references that are in the map. First posted on arXiv 20 Mar 2021.
Builds on
31 of 38 references.
References · 31
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2019The Curious Case of Neural Text Degenerationcited by 2,881
- 2016Visual Dialogcited by 377
- 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
- 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
- 2015A Neural Conversational Modelcited by 195
- 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
- 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
- 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 112
- 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
- 2017Adversarial Learning for Neural Dialogue Generationcited by 103
- 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
19 more
- 2017Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learningcited by 70
- 2017Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generationcited by 59
- 2017Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddingscited by 59
- 2019The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialoguecited by 46
- 2019Multi-step Reasoning via Recurrent Dual Attention for Visual Dialogcited by 41
- 2019Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generationcited by 36
- 2019A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Contextcited by 35
- 2019Improving Generative Visual Dialog by Answering Diverse Questionscited by 21
- 2017End-to-end optimization of goal-driven and visually grounded dialogue systemscited by 17
- 2018Topic-based Evaluation for Conversational Botscited by 16
- 2019Self-Supervised Dialogue Learningcited by 16
- 2018Beyond task success: A closer look at jointly learning to see, ask, and GuessWhatcited by 15
- 2019What Should I Ask? Using Conversationally Informative Rewards for Goal-Oriented Visual Dialogcited by 14
- 2019Visual Dialogue State Tracking for Question Generationcited by 13
- 2017Adversarial Evaluation of Dialogue Modelscited by 12
- 2018Multimodal Hierarchical Reinforcement Learning Policy for Task-Oriented Visual Dialogcited by 8
- 2019Building Task-Oriented Visual Dialog Systems Through Alternative Optimization Between Dialog Policy and Language Generationcited by 6
- 2018What's to know? Uncertainty as a Guide to Asking Goal-oriented Questionscited by 5
- 2019Large-Scale Answerer in Questioner's Mind for Visual Dialog Question Generationcited by 3
Built on by
Citing papers per year: 2021: 1; 2022: 1.
2 papers.