# Building an Efficient and Effective Retrieval-based Dialogue System via Mutual Learning

arXiv: https://arxiv.org/abs/2110.00159 · HTML: https://mlsanity.com/paper/2110.00159/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020)
- [Multi-Stage Document Ranking with BERT](https://mlsanity.com/paper/1910.14424/) (2019)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems](https://mlsanity.com/paper/1506.08909/) (2015)
- [Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval](https://mlsanity.com/paper/2009.12756/) (2020)
- [Pre-training Tasks for Embedding-based Large-scale Retrieval](https://mlsanity.com/paper/2002.03932/) (2020)
- [Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)](https://mlsanity.com/paper/1405.5869/) (2014)
- [Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots](https://mlsanity.com/paper/1612.01627/) (2016)
- [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://mlsanity.com/paper/1911.03688/) (2019)
- [From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbots](https://mlsanity.com/paper/1801.01957/) (2018)
- [Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots](https://mlsanity.com/paper/2004.03588/) (2020)
- [Conversational AI: The Science Behind the Alexa Prize](https://mlsanity.com/paper/1801.03604/) (2018)
- [Training Neural Response Selection for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1906.01543/) (2019)
- [ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation](https://mlsanity.com/paper/1907.05339/) (2019)
- [AliMe Assist: An Intelligent Assistant for Creating an Innovative E-commerce Experience](https://mlsanity.com/paper/1801.05032/) (2018)
- [Multi-turn Response Selection using Dialogue Dependency Relations](https://mlsanity.com/paper/2010.01502/) (2020)
- [An Information Retrieval Approach to Short Text Conversation](https://mlsanity.com/paper/1408.6988/) (2014)
- [Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots](https://mlsanity.com/paper/1901.01824/) (2019)
- [Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently](https://mlsanity.com/paper/2010.10469/) (2020)
- [Syntax-based Deep Matching of Short Texts](https://mlsanity.com/paper/1503.02427/) (2015)
- [Distilling Knowledge for Fast Retrieval-based Chat-bots](https://mlsanity.com/paper/2004.11045/) (2020)

## Built on by (most cited first, 3 of 3)

- [Medical Dialogue: A Survey of Categories, Methods, Evaluation and Challenges](https://mlsanity.com/paper/2405.10630/) (2024)
- [Sparse and Dense Approaches for the Full-rank Retrieval of Responses for Dialogues](https://mlsanity.com/paper/2204.10558/) (2022)
- [Balancing Knowledge Delivery and Emotional Comfort in Healthcare Conversational Systems](https://mlsanity.com/paper/2506.13692/) (2025)

## Abstract (arXiv)

Establishing retrieval-based dialogue systems that can select appropriate responses from the pre-built index has gained increasing attention from researchers. For this task, the adoption of pre-trained language models (such as BERT) has led to remarkable progress in a number of benchmarks. There exist two common approaches, including cross-encoders which perform full attention over the inputs, and bi-encoders that encode the context and response separately. The former gives considerable improvements in accuracy but is often inapplicable in practice for large-scale retrieval given the cost of the full attention required for each sample at test time. The latter is efficient for billions of indexes but suffers from sub-optimal performance. In this work, we propose to combine the best of both worlds to build a retrieval system. Specifically, we employ a fast bi-encoder to replace the traditional feature-based pre-retrieval model (such as BM25) and set the response re-ranking model as a more complicated architecture (such as cross-encoder). To further improve the effectiveness of our framework, we train the pre-retrieval model and the re-ranking model at the same time via mutual learning, which enables two models to learn from each other throughout the training process. We conduct experiments on two benchmarks and evaluation results demonstrate the efficiency and effectiveness of our proposed framework.