# Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversation

arXiv: https://arxiv.org/abs/2108.12582 · HTML: https://mlsanity.com/paper/2108.12582/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 62 references that are in the map.
- First posted on arXiv 28 Aug 2021.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [Towards Deep Conversational Recommendations](https://mlsanity.com/paper/1812.07617/) (2018)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://mlsanity.com/paper/1905.01969/) (2019)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [ParlAI: A Dialog Research Software Platform](https://mlsanity.com/paper/1705.06476/) (2017)
- [Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills](https://mlsanity.com/paper/2004.08449/) (2020)
- [Neural Responding Machine for Short-Text Conversation](https://mlsanity.com/paper/1503.02364/) (2015)

## Built on by (most cited first, 4 of 4)

- [Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances](https://mlsanity.com/paper/2204.10825/) (2022)
- [Measuring and Improving Semantic Diversity of Dialogue Generation](https://mlsanity.com/paper/2210.05725/) (2022)
- [Accurate Knowledge Distillation with n-best Reranking](https://mlsanity.com/paper/2305.12057/) (2023)
- [Understanding and Improving the Exemplar-based Generation for Open-domain Conversation](https://mlsanity.com/paper/2112.06723/) (2021)

## Abstract (arXiv)

Despite the remarkable performance of large-scale generative models in open-domain conversation, they are known to be less practical for building real-time conversation systems due to high latency. On the other hand, retrieval models could return responses with much lower latency but show inferior performance to the large-scale generative models since the conversation quality is bounded by the pre-defined response set. To take advantage of both approaches, we propose a new training method called G2R (Generative-to-Retrieval distillation) that preserves the efficiency of a retrieval model while leveraging the conversational ability of a large-scale generative model by infusing the knowledge of the generative model into the retrieval model. G2R consists of two distinct techniques of distillation: the data-level G2R augments the dialogue dataset with additional responses generated by the large-scale generative model, and the model-level G2R transfers the response quality score assessed by the generative model to the score of the retrieval model by the knowledge distillation loss. Through extensive experiments including human evaluation, we demonstrate that our retrieval-based conversation system trained with G2R shows a substantially improved performance compared to the baseline retrieval model while showing significantly lower inference latency than the large-scale generative models.