mlsanityData to 7 Oct 2026

An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets

arXiv 2201.06219 ↗First posted 17 Jan 2022cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Open-domain dialogue systems aim to converse with humans through text, and dialogue research has heavily relied on benchmark datasets. In this work, we observe the overlapping problem in DailyDialog and OpenSubtitles, two popular open-domain dialogue benchmark datasets. Our systematic analysis then shows that such overlapping can be exploited to obtain fake state-of-the-art performance. Finally, we address this issue by cleaning these datasets and setting up a proper data processing procedure for future research.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 17 Jan 2022.

Builds on

30 of 47 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  9. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  11. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  12. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
18 more
  1. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  2. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  3. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  4. 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
  5. 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
  6. 2016Language to Logical Form with Neural Attentioncited by 158
  7. 2015Neural Responding Machine for Short-Text Conversationcited by 124
  8. 2017Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systemscited by 84
  9. 2020CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Datasetcited by 66
  10. 2020DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterancescited by 36
  11. 2016Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversationcited by 35
  12. 2020Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweightcited by 34
  13. 2021Diversifying Dialog Generation via Adaptive Label Smoothingcited by 25
  14. 2020Discrete Optimization for Unsupervised Sentence Summarization with Word-Level Extractioncited by 23
  15. 2021Generating Relevant and Coherent Dialogue Responses using Self-separated Conditional Variational AutoEncoderscited by 18
  16. 2021Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Trainingcited by 13
  17. 2020Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue Generationcited by 6
  18. 2021A Globally Normalized Neural Model for Semantic Parsingcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 1; 2024: 3; 2025: 1.

8 papers.

  1. 2024Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devicescited by 17
  2. 2022An Equal-Size Hard EM Algorithm for Diverse Dialogue Generationcited by 11
  3. 2022Teacher Forcing Recovers Reward Functions for Text Generationcited by 11
  4. 2023On Evaluation of Document Classification using RVL-CDIPcited by 5
  5. 2024LLMR: Knowledge Distillation with a Large Language Model-Induced Rewardcited by 3
  6. 2022CORAL: Contextual Response Retrievability Loss Function for Training Dialog Generation Modelscited by 1
  7. 2025Chat-Ghosting: A Comparative Study of Methods for Auto-Completion in Dialog Systemscited by 1
  8. 2024Socio-Emotional Response Generation: A Human Evaluation Protocol for LLM-Based Conversational Systemscited by 0