mlsanityData to 7 Oct 2026

Reliable, Adaptable, and Attributable Language Models with Retrieval

arXiv 2403.03187 ↗First posted 5 Mar 2024cs.CL cs.AI cs.LGcited by 73

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Parametric language models (LMs), which are trained on vast amounts of web data, exhibit remarkable flexibility and capability. However, they still face practical challenges such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. In this position paper, we advocate for retrieval-augmented LMs to replace parametric LMs as the next generation of LMs. By incorporating large-scale datastores during inference, retrieval-augmented LMs can be more reliable, adaptable, and attributable. Despite their potential, retrieval-augmented LMs have yet to be widely adopted due to several obstacles: specifically, current retrieval-augmented LMs struggle to leverage helpful text beyond knowledge-intensive tasks such as question answering, have limited interaction between retrieval and LM components, and lack the infrastructure for scaling. To address these, we propose a roadmap for developing general-purpose retrieval-augmented LMs. This involves a reconsideration of datastores and retrievers, the exploration of pipelines with improved retriever-LM interaction, and significant investment in infrastructure for efficient training and inference.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 123 of its 140 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

123 of 140 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining11 Oct
  2. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context2 Oct
  3. 2023BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models2 Oct
  4. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval23 Jun
  5. 2023Active Retrieval Augmented Generation11 May
  6. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study13 Apr
  7. 2023In-Context Retrieval-Augmented Language Models31 Jan
  8. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories20 Dec
  10. 2022Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages19 Dec
  11. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  12. 2022kNN-Prompt: Nearest Neighbor Zero-Shot Inference27 May
2 more
  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 109

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
38 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2017Billion-scale similarity search with GPUscited by 2,477
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  13. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  14. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  15. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  16. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  17. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  18. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  19. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  20. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  21. 2022Red Teaming Language Models with Language Modelscited by 983
  22. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  23. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  24. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  25. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  26. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  27. 2022Galactica: A Large Language Model for Sciencecited by 736
  28. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  29. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  30. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  31. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  32. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  33. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  34. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  35. 2019ELI5: Long Form Question Answeringcited by 567
  36. 2021Editing Factual Knowledge in Language Modelscited by 548
  37. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  38. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454

50 most cited of 109 All 109 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 34; 2025: 32; 2026: 7.

73 papers.

Sort:

By area:

  1. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  2. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  3. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  4. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  5. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 89
  6. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 69
  7. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 54
  8. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  9. 2024How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?cited by 44
  10. 2024Current state of LLM Risks and AI Guardrailscited by 44
  11. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Frameworkcited by 39
  12. 2024RAR-b: Reasoning as Retrieval Benchmarkcited by 33
38 more
  1. 2024GSCo: Towards Generalizable AI in Medicine via Generalist-Specialist Collaborationcited by 33
  2. 2024RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewardscited by 33
  3. 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrievalcited by 33
  4. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
  5. 2024VISA: Retrieval Augmented Generation with Visual Source Attributioncited by 28
  6. 2025RaDeR: Reasoning-aware Dense Retrieval Modelscited by 27
  7. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attributioncited by 26
  8. 2024Context Embeddings for Efficient Answer Generation in RAGcited by 26
  9. 2024To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contextscited by 25
  10. 2024RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructionscited by 25
  11. 2025Provence: efficient and robust context pruning for retrieval-augmented generationcited by 25
  12. 2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attackscited by 24
  13. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  14. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  15. 2024AmbigDocs: Reasoning across Documents on Different Entities under the Same Namecited by 21
  16. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  17. 2025Citations and Trust in LLM Generated Responsescited by 17
  18. 2024Verifiable by Design: Aligning Language Models to Quote from Pre-Training Datacited by 13
  19. 2025Benchmarking Retrieval-Augmented Generation in Multi-Modal Contextscited by 12
  20. 2025TracLLM: A Generic Framework for Attributing Long Context LLMscited by 11
  21. 2025Context-Aware Hierarchical Merging for Long Document Summarizationcited by 9
  22. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  23. 2025Controlled Retrieval-augmented Context Evaluation for Long-form RAGcited by 9
  24. 2024Great Memory, Shallow Reasoning: Limits of kNN-LMscited by 8
  25. 2024Can LLMs Help Create Grammar?: Automating Grammar Creation for Endangered Languages with In-Context Learningcited by 8
  26. 2025Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Modelscited by 8
  27. 2025Evaluating Long-Term Memory for Long-Context Question Answeringcited by 8
  28. 2024Improve Dense Passage Retrieval with Entailment Tuningcited by 7
  29. 2025RankCoT: Refining Knowledge for Retrieval-Augmented Generation through Ranking Chain-of-Thoughtscited by 6
  30. 2024RAC: Efficient LLM Factuality Correction with Retrieval Augmentationcited by 5
  31. 2025AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruptioncited by 5
  32. 2024Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learningcited by 4
  33. 2024Context-Augmented Code Generation Using Programming Knowledge Graphscited by 4
  34. 2024Neurosymbolic AI approach to Attribution in Large Language Modelscited by 3
  35. 2025ConfRAG: Confidence-Guided Retrieval-Augmenting Generationcited by 3
  36. 2025CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluationcited by 3
  37. 2024Data-Centric AI in the Age of Large Language Modelscited by 2
  38. 2025K-COMP: Retrieval-Augmented Medical Domain Question Answering With Knowledge-Injected Compressorcited by 2

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Reimagining Retrieval Augmented Language Models for Answering Queries
  2. 2023BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer
  3. 2023RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models
  4. 2024Unraveling and Mitigating Retriever Inconsistencies in Retrieval-Augmented Large Language Models
  5. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing
  6. 2025Do Retrieval-Augmented Language Models Adapt to Varying User Needs?
  7. 2023Query Rewriting for Retrieval-Augmented Large Language Models
  8. 2024Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Models