# Retrieval-Augmented Transformer-XL for Close-Domain Dialog Generation

arXiv: https://arxiv.org/abs/2105.09235 · HTML: https://mlsanity.com/paper/2105.09235/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 16 of its 16 references that are in the map.
- First posted on arXiv 19 May 2021.

## Placed in

- Retrieval-augmented generation / [Conversational and speech RAG](https://mlsanity.com/rag/conversational-and-speech-rag/)

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset](https://mlsanity.com/paper/1909.05358/) (2019)
- [Hierarchical Recurrent Attention Network for Response Generation](https://mlsanity.com/paper/1701.07149/) (2017)
- [Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation](https://mlsanity.com/paper/1606.00776/) (2016)

## Built on by (most cited first, 7 of 7)

- [Retrieval-Augmented Diffusion Models for Time Series Forecasting](https://mlsanity.com/paper/2410.18712/) (2024)
- [Accelerating Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.15246/) (2024)
- [MQRetNN: Multi-Horizon Time Series Forecasting with Retrieval Augmentation](https://mlsanity.com/paper/2207.10517/) (2022)
- [Semi-Parametric Inducing Point Networks and Neural Processes](https://mlsanity.com/paper/2205.11718/) (2022)
- [RATSF: Empowering Customer Service Volume Management through Retrieval-Augmented Time-Series Forecasting](https://mlsanity.com/paper/2403.04180/) (2024)
- [Retrieval-Augmented Neural Response Generation Using Logical Reasoning and Relevance Scoring](https://mlsanity.com/paper/2310.13566/) (2023)
- [LLM-Based Insight Extraction for Contact Center Analytics and Cost-Efficient Deployment](https://mlsanity.com/paper/2503.19090/) (2025)

## Abstract (arXiv)

Transformer-based models have demonstrated excellent capabilities of capturing patterns and structures in natural language generation and achieved state-of-the-art results in many tasks. In this paper we present a transformer-based model for multi-turn dialog response generation. Our solution is based on a hybrid approach which augments a transformer-based generative model with a novel retrieval mechanism, which leverages the memorized information in the training data via k-Nearest Neighbor search. Our system is evaluated on two datasets made by customer/assistant dialogs: the Taskmaster-1, released by Google and holding high quality, goal-oriented conversational data and a proprietary dataset collected from a real customer service call center. Both achieve better BLEU scores over strong baselines.