mlsanityData to 7 Oct 2026

Generate rather than Retrieve: Large Language Models are Strong Context Generators

arXiv 2209.10063 ↗First posted 21 Sep 2022cs.CL cs.AIcited by 296

Not on the map. Why?

Cited by 82 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Knowledge-intensive tasks, such as open-domain question answering (QA), require access to a large amount of world or domain knowledge. A common approach for knowledge-intensive tasks is to employ a retrieve-then-read pipeline that first retrieves a handful of relevant contextual documents from an external corpus such as Wikipedia and then predicts an answer conditioned on the retrieved documents. In this paper, we present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators. We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer. Furthermore, we propose a novel clustering-based prompting method that selects distinct prompts, resulting in the generated documents that cover different perspectives, leading to better recall over acceptable answers. We conduct extensive experiments on three different knowledge-intensive tasks, including open-domain QA, fact checking, and dialogue system. Notably, GenRead achieves 71.6 and 54.4 exact match scores on TriviaQA and WebQ, significantly outperforming the state-of-the-art retrieve-then-read pipeline DPR-FiD by +4.0 and +3.9, without retrieving any documents from any external knowledge source. Lastly, we demonstrate the model performance can be further improved by combining retrieval and generation. Our code and generated documents can be found at https://github.com/wyu97/GenRead.

Key facts. Cited by 296 papers in the map (data through 7 Oct 2026). Builds on 55 of its 66 references that are in the map. First posted on arXiv 21 Sep 2022.

Builds on

55 of 66 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  12. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  13. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  14. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  15. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  16. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  17. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  18. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  19. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  20. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  21. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  22. 2020Autoregressive Entity Retrievalcited by 333
  23. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  24. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  25. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  26. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 213
  27. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  28. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  29. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  30. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  31. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  32. 2022Faithful Reasoning Using Large Language Modelscited by 130
  33. 2020A Survey of Knowledge-Enhanced Text Generationcited by 119
  34. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
  35. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  36. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71
  37. 2021KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answeringcited by 66
  38. 2021Fool Me Twice: Entailment from Wikipedia Gamificationcited by 50

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 93; 2024: 121; 2025: 52; 2026: 19.

296 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  4. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  5. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  6. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  7. 2023Active Retrieval Augmented Generationcited by 506
  8. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  9. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  10. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  11. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  12. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
38 more
  1. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  2. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  3. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  4. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  5. 2023Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender Systemcited by 241
  6. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  7. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  8. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  9. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  10. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 180
  11. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  12. 2023Knowledge Graph Prompting for Multi-Document Question Answeringcited by 150
  13. 2023Query2doc: Query Expansion with Large Language Modelscited by 146
  14. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 146
  15. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  16. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 124
  17. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 123
  18. 2023Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!cited by 119
  19. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 119
  20. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  21. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memorycited by 103
  22. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
  23. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  24. 2024Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Modelscited by 93
  25. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  26. 2023Dense X Retrieval: What Retrieval Granularity Should We Use?cited by 88
  27. 2023Improving Language Models via Plug-and-Play Retrieval Feedbackcited by 87
  28. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 87
  29. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  30. 2023Towards Mitigating Hallucination in Large Language Models via Self-Reflectioncited by 81
  31. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  32. 2023Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-Incited by 71
  33. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  34. 2023MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Taskscited by 66
  35. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 66
  36. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  37. 2024KnowAgent: Knowledge-Augmented Planning for LLM-Based Agentscited by 61
  38. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59

50 most cited of 296 All 296 (JSON)