# Hybrid Generative-Retrieval Transformers for Dialogue Domain Adaptation

arXiv: https://arxiv.org/abs/2003.01680 · HTML: https://mlsanity.com/paper/2003.01680/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2020.

## Built on by (most cited first, 5 of 5)

- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [AARGH! End-to-end Retrieval-Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2209.03632/) (2022)
- [A Student-Teacher Architecture for Dialog Domain Adaptation under the Meta-Learning Setting](https://mlsanity.com/paper/2104.02689/) (2021)
- [Towards a Universal NLG for Dialogue Systems and Simulators with Future Bridging](https://mlsanity.com/paper/2105.10267/) (2021)

## Abstract (arXiv)

Domain adaptation has recently become a key problem in dialogue systems research. Deep learning, while being the preferred technique for modeling such systems, works best given massive training data. However, in the real-world scenario, such resources aren't available for every new domain, so the ability to train with a few dialogue examples can be considered essential. Pre-training on large data sources and adapting to the target data has become the standard method for few-shot problems within the deep learning framework. In this paper, we present the winning entry at the fast domain adaptation task of DSTC8, a hybrid generative-retrieval model based on GPT-2 fine-tuned to the multi-domain MetaLWOz dataset. Robust and diverse in response generation, our model uses retrieval logic as a fallback, being SoTA on MetaLWOz in human evaluation (>4% improvement over the 2nd place system) and attaining competitive generalization performance in adaptation to the unseen MultiWOZ dataset.