mlsanityData to 7 Oct 2026

Retrieval-Augmented Transformer-XL for Close-Domain Dialog Generation

arXiv 2105.09235 ↗First posted 19 May 2021cs.CL cs.AIcited by 7

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

Transformer-based models have demonstrated excellent capabilities of capturing patterns and structures in natural language generation and achieved state-of-the-art results in many tasks. In this paper we present a transformer-based model for multi-turn dialog response generation. Our solution is based on a hybrid approach which augments a transformer-based generative model with a novel retrieval mechanism, which leverages the memorized information in the training data via k-Nearest Neighbor search. Our system is evaluated on two datasets made by customer/assistant dialogs: the Taskmaster-1, released by Google and holding high quality, goal-oriented conversational data and a proprietary dataset collected from a real customer service call center. Both achieve better BLEU scores over strong baselines.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 16 of its 16 references that are in the map. First posted on arXiv 19 May 2021.

Builds on

16 of 16 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  4. 2017Billion-scale similarity search with GPUscited by 2,477
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  9. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  10. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  11. 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
  12. 2015A Neural Conversational Modelcited by 195
4 more
  1. 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188
  2. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 139
  3. 2017Hierarchical Recurrent Attention Network for Response Generationcited by 34
  4. 2016Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generationcited by 16

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 3; 2025: 1.

7 papers.

  1. 2024Retrieval-Augmented Diffusion Models for Time Series Forecastingcited by 35
  2. 2024Accelerating Retrieval-Augmented Generationcited by 14
  3. 2022MQRetNN: Multi-Horizon Time Series Forecasting with Retrieval Augmentationcited by 9
  4. 2022Semi-Parametric Inducing Point Networks and Neural Processescited by 6
  5. 2024RATSF: Empowering Customer Service Volume Management through Retrieval-Augmented Time-Series Forecastingcited by 2
  6. 2023Retrieval-Augmented Neural Response Generation Using Logical Reasoning and Relevance Scoringcited by 1
  7. 2025LLM-Based Insight Extraction for Contact Center Analytics and Cost-Efficient Deploymentcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022AARGH! End-to-end Retrieval-Generation for Task-Oriented Dialog
  2. 2021Efficient Retrieval Augmented Generation from Unstructured Knowledge for Task-Oriented Dialog
  3. 2024Retrieval Augmented End-to-End Spoken Dialog Models
  4. 2022A Survey on Retrieval-Augmented Text Generation
  5. 2025Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation
  6. 2022Logical Reasoning for Task Oriented Dialogue Systems
  7. 2026Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment
  8. 2022KGI: An Integrated Framework for Knowledge Intensive Language Tasks