# Can Visual Dialogue Models Do Scorekeeping? Exploring How Dialogue Representations Incrementally Encode Shared Knowledge

arXiv: https://arxiv.org/abs/2204.06970 · HTML: https://mlsanity.com/paper/2204.06970/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 55 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [Higher-order Coreference Resolution with Coarse-to-fine Inference](https://mlsanity.com/paper/1804.05392/) (2018)
- [Dialogue Natural Language Inference](https://mlsanity.com/paper/1811.00671/) (2018)
- [Transforming Question Answering Datasets Into Natural Language Inference Datasets](https://mlsanity.com/paper/1809.02922/) (2018)
- [A Neural Network Approach to Context-Sensitive Generation of Conversational Responses](https://mlsanity.com/paper/1506.06714/) (2015)
- [FOIL it! Find One mismatch between Image and Language caption](https://mlsanity.com/paper/1705.01359/) (2017)
- [Evaluating Coherence in Dialogue Systems using Entailment](https://mlsanity.com/paper/1904.03371/) (2019)
- [Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study](https://mlsanity.com/paper/1906.01603/) (2019)
- [History for Visual Dialog: Do we really need it?](https://mlsanity.com/paper/2005.07493/) (2020)
- [Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA](https://mlsanity.com/paper/2103.09591/) (2021)
- [Improving Generative Visual Dialog by Answering Diverse Questions](https://mlsanity.com/paper/1909.10470/) (2019)
- [Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat](https://mlsanity.com/paper/1809.03408/) (2018)
- [Probing Task-Oriented Dialogue Representation from Language Models](https://mlsanity.com/paper/2010.13912/) (2020)
- [Grounded Agreement Games: Emphasizing Conversational Grounding in Visual Dialogue Settings](https://mlsanity.com/paper/1908.11279/) (2019)
- [Probing Neural Dialog Models for Conversational Understanding](https://mlsanity.com/paper/2006.08331/) (2020)
- [Analysing the potential of seq-to-seq models for incremental interpretation in task-oriented dialogue](https://mlsanity.com/paper/1808.09178/) (2018)
- [How To Evaluate Your Dialogue System: Probe Tasks as an Alternative for Token-level Evaluation Metrics](https://mlsanity.com/paper/2008.10427/) (2020)

## Built on by (most cited first, 2 of 2)

- [Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents](https://mlsanity.com/paper/2305.13455/) (2023)
- [Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategy](https://mlsanity.com/paper/2304.07007/) (2023)

## Abstract (arXiv)

Cognitively plausible visual dialogue models should keep a mental scoreboard of shared established facts in the dialogue context. We propose a theory-based evaluation method for investigating to what degree models pretrained on the VisDial dataset incrementally build representations that appropriately do scorekeeping. Our conclusion is that the ability to make the distinction between shared and privately known statements along the dialogue is moderately present in the analysed models, but not always incrementally consistent, which may partially be due to the limited need for grounding interactions in the original task.