mlsanityData to 7 Oct 2026

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

arXiv 2005.11401 ↗First posted 22 May 2020cs.CL cs.LGcited by 9,160

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.

Key facts. Cited by 9,160 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 54 of its 67 references that are in the map. First posted on arXiv 22 May 2020.

Builds on

54 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
38 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  4. 2018Hierarchical Neural Story Generationcited by 1,182
  5. 2017Mixed Precision Trainingcited by 1,135
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  7. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  9. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  10. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  11. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  12. 2019Passage Re-ranking with BERTcited by 629
  13. 2019ELI5: Long Form Question Answeringcited by 567
  14. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  15. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  16. 2014Memory Networkscited by 354
  17. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  18. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 221
  19. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 194
  20. 2020How Context Affects Language Models' Factual Predictionscited by 181
  21. 2017A Knowledge-Grounded Neural Conversation Modelcited by 136
  22. 2017Simple and Effective Multi-Paragraph Reading Comprehensioncited by 120
  23. 2019Large Memory Layers with Product Keyscited by 120
  24. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  25. 2017Generating Sentences by Editing Prototypescited by 88
  26. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  27. 2018Retrieve and Refine: Improved Sequence Generation Models For Dialoguecited by 82
  28. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  29. 2018A Retrieve-and-Edit Framework for Predicting Structured Outputscited by 73
  30. 2019Reasoning Over Semantic-Level Graph for Fact Checkingcited by 69
  31. 2019Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answeringcited by 63
  32. 2018Towards Exploiting Background Knowledge for Building Conversation Systemscited by 57
  33. 2018Towards a Better Metric for Evaluating Question Generation Systemscited by 52
  34. 2020PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generationcited by 42
  35. 2017When Will AI Exceed Human Performance? Evidence from AI Expertscited by 41
  36. 2019How Decoding Strategies Affect the Verifiability of Generated Textcited by 27
  37. 2019Finding Generalizable Evidence by Learning to Convince Q&A Modelscited by 25
  38. 2017Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answeringcited by 24

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 159; 2022: 229; 2023: 519; 2024: 1986; 2025: 2764; 2026: 3503.

9,160 papers.

Sort:

By area:

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  10. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  11. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
38 more
  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  4. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  11. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
  12. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  13. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  14. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  15. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  16. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  17. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  18. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  19. 2022Galactica: A Large Language Model for Sciencecited by 736
  20. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  21. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  22. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  23. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  24. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  25. 2024A Survey on Large Language Models for Code Generationcited by 646
  26. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  27. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  28. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  29. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  30. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  31. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  32. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  33. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  34. 2023Active Retrieval Augmented Generationcited by 506
  35. 2024Large Language Models: A Surveycited by 506
  36. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  37. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  38. 2022Memory-Based Model Editing at Scalecited by 443

50 most cited of 9,160 All 9,160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Dynamic Parametric Retrieval Augmented Generation for Test-time Knowledge Enhancement
  2. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  3. 2024Empowering Large Language Models to Set up a Knowledge Retrieval Indexer via Self-Learning
  4. 2024On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models
  5. 2025Zero-RAG: Towards Retrieval-Augmented Generation with Zero Redundant Knowledge
  6. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
  7. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  8. 2024KBM: Delineating Knowledge Boundary for Adaptive Retrieval in Large Language Models