An Evaluation Dataset and Strategy for Building Robust Multi-turn Response Selection Model
Not on the map. Why?
Abstract (arXiv)
Multi-turn response selection models have recently shown comparable performance to humans in several benchmark datasets. However, in the real environment, these models often have weaknesses, such as making incorrect predictions based heavily on superficial patterns without a comprehensive understanding of the context. For example, these models often give a high score to the wrong response candidate containing several keywords related to the context but using the inconsistent tense. In this study, we analyze the weaknesses of the open-domain Korean Multi-turn response selection models and publish an adversarial dataset to evaluate these weaknesses. We also suggest a strategy to build a robust model in this adversarial environment.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 26 references that are in the map. First posted on arXiv 10 Sep 2021.
Builds on
23 of 26 references.
References · 23
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
- 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
- 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
- 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
- 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
- 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 280
- 2018Stress Test Evaluation for Natural Language Inferencecited by 213
- 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 179
- 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172
- 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
11 more
- 2019Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoringcited by 165
- 2020Learning from Failure: Training Debiased Classifier from Biased Classifiercited by 133
- 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
- 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 115
- 2016Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbotscited by 103
- 2020Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performancecited by 85
- 2020Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbotscited by 73
- 2018Modeling Multi-turn Conversation with Deep Utterance Aggregationcited by 65
- 2020Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selectioncited by 36
- 2020Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialoguescited by 34
- 2019TripleNet: Triple Attention Network for Multi-Turn Response Selection in Retrieval-based Chatbotscited by 4
Built on by
Citing papers per year: 2022: 1.
1 paper.