mlsanityData to 7 Oct 2026

RetGen: A Joint framework for Retrieval and Grounded Text Generation Modeling

arXiv 2105.06597 ↗First posted 14 May 2021cs.CL cs.AIcited by 22

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Recent advances in large-scale pre-training such as GPT-3 allow seemingly high quality text to be generated from a given prompt. However, such generation systems often suffer from problems of hallucinated facts, and are not inherently designed to incorporate useful external information. Grounded generation models appear to offer remedies, but their training typically relies on rarely-available parallel data where information-relevant documents are provided for context. We propose a framework that alleviates this data constraint by jointly training a grounded generator and document retriever on the language model signal. The model learns to reward retrieval of the documents with the highest utility in generation, and attentively combines them using a Mixture-of-Experts (MoE) ensemble to generate follow-on text. We demonstrate that both generator and retriever can take advantage of this joint training and work synergistically to produce more informative and relevant text in both prose and dialogue generation.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 38 of its 53 references that are in the map. First posted on arXiv 14 May 2021.

Builds on

38 of 53 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  8. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  9. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  10. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
26 more
  1. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  2. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  3. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  4. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  5. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  6. 2017A Knowledge-Grounded Neural Conversation Modelcited by 136
  7. 2018Generating Informative and Diverse Conversational Responses via Adversarial Information Maximizationcited by 114
  8. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  9. 2020Knowledge-Grounded Dialogue Generation with Pre-trained Language Modelscited by 107
  10. 2018A Dataset for Document Grounded Conversationscited by 104
  11. 2019On the Use of ArXiv as a Datasetcited by 101
  12. 2020Pre-training via Paraphrasingcited by 92
  13. 2018A Retrieve-and-Edit Framework for Predicting Structured Outputscited by 73
  14. 2020SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teachingcited by 71
  15. 2020Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Rewardcited by 66
  16. 2019Question Answering as an Automatic Evaluation Metric for News Article Summarizationcited by 63
  17. 2018Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memorycited by 49
  18. 2020A Controllable Model of Grounded Response Generationcited by 49
  19. 2018Response Generation by Context-aware Prototype Editingcited by 40
  20. 2021Efficient Retrieval Augmented Generation from Unstructured Knowledge for Task-Oriented Dialogcited by 33
  21. 2019Conversing by Reading: Contentful Neural Conversation with On-demand Machine Readingcited by 32
  22. 2019Text Generation with Exemplar-based Adaptive Decodingcited by 28
  23. 2017Flexible End-to-End Dialogue System for Knowledge Grounded Conversationcited by 18
  24. 2019A Hybrid Retrieval-Generation Neural Conversation Modelcited by 18
  25. 2019Contrastive Multi-document Question Generationcited by 15
  26. 2020Robust Conversational AI with Grounded Text Generationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 7; 2024: 3; 2025: 7; 2026: 2.

22 papers.

Sort:

By area:

  1. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  2. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  3. 2022Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Datacited by 61
  4. 2024Jill Watson: A Virtual Teaching Assistant powered by ChatGPTcited by 24
  5. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  6. 2024Class-Level Code Generation from Natural Language Using Iterative, Tool-Enhanced Reasoning over Repositorycited by 14
  7. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  8. 2023Diversify Question Generation with Retrieval-Augmented Style Transfercited by 10
  9. 2023RRAML: Reinforced Retrieval Augmented Machine Learningcited by 6
  10. 2025Exp4Fuse: A Rank Fusion Framework for Enhanced Sparse Retrieval using Large Language Model-based Query Expansioncited by 6
  11. 2025The Behavior Gap: Evaluating Zero-shot LLM Agents in Complex Task-Oriented Dialogscited by 6
  12. 2022Incorporating Task-specific Concept Knowledge into Script Learningcited by 3
10 more
  1. 2022KPT: Keyword-guided Pre-training for Grounded Dialog Generationcited by 3
  2. 2025Through the Prism of Culture: Evaluating LLMs' Understanding of Indian Subcultures and Traditionscited by 3
  3. 2025Filtering with Confidence: When Data Augmentation Meets Conformal Predictioncited by 2
  4. 2023Dynamic Retrieval-Augmented Generationcited by 1
  5. 2025Improving End-to-End Training of Retrieval-Augmented Generation Models via Joint Stochastic Approximationcited by 1
  6. 2023Hexa: Self-Improving for Knowledge-Grounded Dialogue Systemcited by 0
  7. 2023An Initialization Schema for Neuronal Networks on Tabular Datacited by 0
  8. 2025Retrieval-Constrained Decoding Reveals Underestimated Parametric Knowledge in Language Modelscited by 0
  9. 2026BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answeringcited by 0
  10. 2026Structures Facilitate Retrieve, Rerank, and Generatecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Retrieval is Accurate Generation
  2. 2023Active Retrieval Augmented Generation
  3. 2025Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models
  4. 2022A Survey on Retrieval-Augmented Text Generation
  5. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memory
  6. 2023Prompt Generate Train (PGT): Few-shot Domain Adaption of Retrieval Augmented Generation Models for Open Book Question-Answering
  7. 2022Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling
  8. 2025From Retrieval to Generation: Comparing Different Approaches