mlsanityData to 7 Oct 2026

REALM: Retrieval-Augmented Language Model Pre-Training

arXiv 2002.08909 ↗First posted 10 Feb 2020cs.CL cs.LGcited by 2,126

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.

Key facts. Cited by 2,126 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 43 references that are in the map. First posted on arXiv 10 Feb 2020.

Builds on

31 of 43 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
19 more
  1. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  2. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  3. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  4. 2014Neural Turing Machinescited by 630
  5. 2014Memory Networkscited by 354
  6. 2015Skip-Thought Vectorscited by 321
  7. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  8. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  9. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  10. 2015Semi-supervised Sequence Learningcited by 198
  11. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  12. 2017Simple and Effective Multi-Paragraph Reading Comprehensioncited by 120
  13. 2019Large Memory Layers with Product Keyscited by 120
  14. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  15. 2017Generating Sentences by Editing Prototypescited by 88
  16. 2019A Discrete Hard EM Approach for Weakly Supervised Question Answeringcited by 78
  17. 2018A Retrieve-and-Edit Framework for Predicting Structured Outputscited by 73
  18. 2019Knowledge Guided Text Retrieval and Reading for Open Domain Question Answeringcited by 59
  19. 2016Learning Recurrent Span Representations for Extractive Question Answeringcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 185; 2022: 252; 2023: 315; 2024: 492; 2025: 432; 2026: 449.

2,126 papers.

Sort:

By area:

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  11. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
38 more
  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  7. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  8. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  9. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  10. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  11. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  12. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  13. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  14. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  15. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  16. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  17. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  18. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  19. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  20. 2023A Comprehensive Overview of Large Language Modelscited by 639
  21. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  22. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  23. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  24. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  25. 2023Active Retrieval Augmented Generationcited by 506
  26. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  27. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  28. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  29. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  30. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  31. 2023Cognitive Architectures for Language Agentscited by 422
  32. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  33. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  34. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  35. 2023Augmented Language Models: a Surveycited by 398
  36. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  37. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  38. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357

50 most cited of 2,126 All 2,126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Teaching Smaller Language Models To Generalise To Unseen Compositional Questions (Full Thesis)
  2. 2024Unraveling and Mitigating Retriever Inconsistencies in Retrieval-Augmented Large Language Models
  3. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Models
  4. 2026When to Retrieve During Reasoning: Adaptive Retrieval for Large Reasoning Models
  5. 2024Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorization
  6. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
  7. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
  8. 2024Synthetic continued pretraining