# Distilling Knowledge for Fast Retrieval-based Chat-bots

arXiv: https://arxiv.org/abs/2004.11045 · HTML: https://mlsanity.com/paper/2004.11045/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2020.

## Built on by (most cited first, 10 of 10)

- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs](https://mlsanity.com/paper/2402.12052/) (2024)
- [Multi-View Feature Representation for Dialogue Generation with Bidirectional Distillation](https://mlsanity.com/paper/2102.10780/) (2021)
- [Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversation](https://mlsanity.com/paper/2108.12582/) (2021)
- [Building an Efficient and Effective Retrieval-based Dialogue System via Mutual Learning](https://mlsanity.com/paper/2110.00159/) (2021)
- [Retrieval-Augmented Generation for Mobile Edge Computing via Large Language Model](https://mlsanity.com/paper/2412.20820/) (2024)
- [Learn What Is Possible, Then Choose What Is Best: Disentangling One-To-Many Relations in Language Through Text-based Games](https://mlsanity.com/paper/2304.07258/) (2023)
- [PaRT: Enhancing Proactive Social Chatbots with Personalized Real-Time Retrieval](https://mlsanity.com/paper/2504.20624/) (2025)
- [Mix-and-Match: Scalable Dialog Response Retrieval using Gaussian Mixture Embeddings](https://mlsanity.com/paper/2204.02710/) (2022)
- [On Elastic Language Models](https://mlsanity.com/paper/2311.07204/) (2023)

## Abstract (arXiv)

Response retrieval is a subset of neural ranking in which a model selects a suitable response from a set of candidates given a conversation history. Retrieval-based chat-bots are typically employed in information seeking conversational systems such as customer support agents. In order to make pairwise comparisons between a conversation history and a candidate response, two approaches are common: cross-encoders performing full self-attention over the pair and bi-encoders encoding the pair separately. The former gives better prediction quality but is too slow for practical use. In this paper, we propose a new cross-encoder architecture and transfer knowledge from this model to a bi-encoder model using distillation. This effectively boosts bi-encoder performance at no cost during inference time. We perform a detailed analysis of this approach on three response retrieval datasets.