# An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets

arXiv: https://arxiv.org/abs/2201.06219 · HTML: https://mlsanity.com/paper/2201.06219/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 17 Jan 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [The Design and Implementation of XiaoIce, an Empathetic Social Chatbot](https://mlsanity.com/paper/1812.08989/) (2018)
- [Language to Logical Form with Neural Attention](https://mlsanity.com/paper/1601.01280/) (2016)
- [Neural Responding Machine for Short-Text Conversation](https://mlsanity.com/paper/1503.02364/) (2015)
- [Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systems](https://mlsanity.com/paper/1704.00057/) (2017)
- [CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset](https://mlsanity.com/paper/2002.11893/) (2020)
- [DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances](https://mlsanity.com/paper/2012.01775/) (2020)
- [Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation](https://mlsanity.com/paper/1607.00970/) (2016)
- [Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweight](https://mlsanity.com/paper/2004.02594/) (2020)
- [Diversifying Dialog Generation via Adaptive Label Smoothing](https://mlsanity.com/paper/2105.14556/) (2021)
- [Discrete Optimization for Unsupervised Sentence Summarization with Word-Level Extraction](https://mlsanity.com/paper/2005.01791/) (2020)
- [Generating Relevant and Coherent Dialogue Responses using Self-separated Conditional Variational AutoEncoders](https://mlsanity.com/paper/2106.03410/) (2021)
- [Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Training](https://mlsanity.com/paper/2105.15171/) (2021)
- [Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue Generation](https://mlsanity.com/paper/2003.00639/) (2020)
- [A Globally Normalized Neural Model for Semantic Parsing](https://mlsanity.com/paper/2106.03376/) (2021)

## Built on by (most cited first, 8 of 8)

- [Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devices](https://mlsanity.com/paper/2406.03777/) (2024)
- [An Equal-Size Hard EM Algorithm for Diverse Dialogue Generation](https://mlsanity.com/paper/2209.14627/) (2022)
- [Teacher Forcing Recovers Reward Functions for Text Generation](https://mlsanity.com/paper/2210.08708/) (2022)
- [On Evaluation of Document Classification using RVL-CDIP](https://mlsanity.com/paper/2306.12550/) (2023)
- [LLMR: Knowledge Distillation with a Large Language Model-Induced Reward](https://mlsanity.com/paper/2409.12500/) (2024)
- [CORAL: Contextual Response Retrievability Loss Function for Training Dialog Generation Models](https://mlsanity.com/paper/2205.10558/) (2022)
- [Chat-Ghosting: A Comparative Study of Methods for Auto-Completion in Dialog Systems](https://mlsanity.com/paper/2507.05940/) (2025)
- [Socio-Emotional Response Generation: A Human Evaluation Protocol for LLM-Based Conversational Systems](https://mlsanity.com/paper/2412.04492/) (2024)

## Abstract (arXiv)

Open-domain dialogue systems aim to converse with humans through text, and dialogue research has heavily relied on benchmark datasets. In this work, we observe the overlapping problem in DailyDialog and OpenSubtitles, two popular open-domain dialogue benchmark datasets. Our systematic analysis then shows that such overlapping can be exploited to obtain fake state-of-the-art performance. Finally, we address this issue by cleaning these datasets and setting up a proper data processing procedure for future research.