# A Systematic Evaluation of Response Selection for Open Domain Dialogue

arXiv: https://arxiv.org/abs/2208.04379 · HTML: https://mlsanity.com/paper/2208.04379/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 8 Aug 2022.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems](https://mlsanity.com/paper/1506.08909/) (2015)
- [The Design and Implementation of XiaoIce, an Empathetic Social Chatbot](https://mlsanity.com/paper/1812.08989/) (2018)
- [Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills](https://mlsanity.com/paper/2004.08449/) (2020)
- [Enabling Language Models to Fill in the Blanks](https://mlsanity.com/paper/2005.05339/) (2020)
- [Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots](https://mlsanity.com/paper/1612.01627/) (2016)
- [Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context](https://mlsanity.com/paper/1911.10484/) (2019)
- [Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots](https://mlsanity.com/paper/2004.03588/) (2020)
- [Dialogue Response Ranking Training with Large-Scale Human Feedback Data](https://mlsanity.com/paper/2009.06978/) (2020)
- [Modeling Multi-turn Conversation with Deep Utterance Aggregation](https://mlsanity.com/paper/1806.09102/) (2018)
- [Conversational AI: The Science Behind the Alexa Prize](https://mlsanity.com/paper/1801.03604/) (2018)
- [Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining](https://mlsanity.com/paper/2009.11321/) (2020)
- [Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References](https://mlsanity.com/paper/1907.10568/) (2019)
- [A Deep Reinforcement Learning Chatbot](https://mlsanity.com/paper/1709.02349/) (2017)
- [Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selection](https://mlsanity.com/paper/2009.04703/) (2020)
- [Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialogues](https://mlsanity.com/paper/2009.06265/) (2020)
- [Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations](https://mlsanity.com/paper/2008.12348/) (2020)
- [Structural Pre-training for Dialogue Comprehension](https://mlsanity.com/paper/2105.10956/) (2021)
- [Are Neural Open-Domain Dialog Systems Robust to Speech Recognition Errors in the Dialog History? An Empirical Study](https://mlsanity.com/paper/2008.07683/) (2020)
- [Policy-Driven Neural Response Generation for Knowledge-Grounded Dialogue Systems](https://mlsanity.com/paper/2005.12529/) (2020)
- [Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation](https://mlsanity.com/paper/2106.05894/) (2021)
- [ISO-Standard Domain-Independent Dialogue Act Tagging for Conversational Agents](https://mlsanity.com/paper/1806.04327/) (2018)
- [Neural Response Ranking for Social Conversation: A Data-Efficient Approach](https://mlsanity.com/paper/1811.00967/) (2018)
- [Alquist 4.0: Towards Social Intelligence Using Generative Models and Dialogue Personalization](https://mlsanity.com/paper/2109.07968/) (2021)
- [Proto: A Neural Cocktail for Generating Appealing Conversations](https://mlsanity.com/paper/2109.02513/) (2021)

## Built on by (most cited first, 4 of 4)

- [A Transformer-based Response Evaluator for Open-Domain Spoken Conversation](https://mlsanity.com/paper/2302.04424/) (2023)
- [Modeling the One-to-Many Property in Open-Domain Dialogue with LLMs](https://mlsanity.com/paper/2506.15131/) (2025)
- [Social Commonsense-Guided Search Query Generation for Open-Domain Knowledge-Powered Conversations](https://mlsanity.com/paper/2310.14340/) (2023)
- [ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents](https://mlsanity.com/paper/2608.21969/) (2026)

## Abstract (arXiv)

Recent progress on neural approaches for language processing has triggered a resurgence of interest on building intelligent open-domain chatbots. However, even the state-of-the-art neural chatbots cannot produce satisfying responses for every turn in a dialog. A practical solution is to generate multiple response candidates for the same context, and then perform response ranking/selection to determine which candidate is the best. Previous work in response selection typically trains response rankers using synthetic data that is formed from existing dialogs by using a ground truth response as the single appropriate response and constructing inappropriate responses via random selection or using adversarial methods. In this work, we curated a dataset where responses from multiple response generators produced for the same dialog context are manually annotated as appropriate (positive) and inappropriate (negative). We argue that such training data better matches the actual use case examples, enabling the models to learn to rank responses effectively. With this new dataset, we conduct a systematic evaluation of state-of-the-art methods for response selection, and demonstrate that both strategies of using multiple positive candidates and using manually verified hard negative candidates can bring in significant performance improvement in comparison to using the adversarial training data, e.g., increase of 3% and 13% in Recall@1 score, respectively.