# The Spot the Difference corpus: a multi-modal corpus of spontaneous task oriented spoken interactions

arXiv: https://arxiv.org/abs/1805.05091 · HTML: https://mlsanity.com/paper/1805.05091/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2018.

## Built on by (most cited first, 5 of 5)

- [Conversational Grounding: Annotation and Analysis of Grounding Acts and Grounding Units](https://mlsanity.com/paper/2403.16609/) (2024)
- [Understanding Common Ground Misalignment in Goal-Oriented Dialog: A Case-Study with Ubuntu Chat Logs](https://mlsanity.com/paper/2503.12370/) (2025)
- [Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Models](https://mlsanity.com/paper/2406.14035/) (2024)
- [Frame of Reference: Addressing the Challenges of Common Ground Representation in Situational Dialogs](https://mlsanity.com/paper/2601.09365/) (2026)
- [Spot the Difference: A Cooperative Object-Referring Game in Non-Perfectly Co-Observable Scene](https://mlsanity.com/paper/2203.08362/) (2022)

## Abstract (arXiv)

This paper describes the Spot the Difference Corpus which contains 54 interactions between pairs of subjects interacting to find differences in two very similar scenes. The setup used, the participants' metadata and details about collection are described. We are releasing this corpus of task-oriented spontaneous dialogues. This release includes rich transcriptions, annotations, audio and video. We believe that this dataset constitutes a valuable resource to study several dimensions of human communication that go from turn-taking to the study of referring expressions. In our preliminary analyses we have looked at task success (how many differences were found out of the total number of differences) and how it evolves over time. In addition we have looked at scene complexity provided by the RGB components' entropy and how it could relate to speech overlaps, interruptions and the expression of uncertainty. We found there is a tendency that more complex scenes have more competitive interruptions.