mlsanityData to 7 Oct 2026

In-Context Retrieval-Augmented Language Models

arXiv 2302.00083 ↗First posted 31 Jan 2023cs.CL cs.IRcited by 694

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance. In addition, they can mitigate the problem of factually inaccurate text generation and provide natural source attribution mechanism. Existing RALM approaches focus on modifying the LM architecture in order to facilitate the incorporation of external information, significantly complicating deployment. This paper considers a simple alternative, which we dub In-Context RALM: leaving the LM architecture unchanged and prepending grounding documents to the input, without any further training of the LM. We show that In-Context RALM that builds on off-the-shelf general purpose retrievers provides surprisingly large LM gains across model sizes and diverse corpora. We also demonstrate that the document retrieval and ranking mechanism can be specialized to the RALM setting to further boost performance. We conclude that In-Context RALM has considerable potential to increase the prevalence of LM grounding, particularly in settings where a pretrained LM must be used without modification or even via API access.

Key facts. Cited by 694 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 36 of its 58 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

36 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  2. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
20 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  3. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  5. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  6. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  7. 2022A Survey on In-context Learningcited by 717
  8. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  9. 2019Defending Against Neural Fake Newscited by 553
  10. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  11. 2019Challenges in Building Intelligent Open-domain Dialog Systemscited by 147
  12. 2022Training Language Models with Memory Augmentationcited by 112
  13. 2021Efficient Nearest Neighbor Language Modelscited by 93
  14. 2022Parallel Context Windows for Large Language Modelscited by 81
  15. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  16. 2021Learning to Retrieve Passages without Supervisioncited by 46
  17. 2021R2-D2: A Modular Baseline for Open-Domain Question Answeringcited by 37
  18. 2021The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Designcited by 33
  19. 2022Standing on the Shoulders of Giant Frozen Language Modelscited by 33
  20. 2022Decoupled Context Processing for Context Augmented Language Modelingcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 93; 2024: 259; 2025: 216; 2026: 126.

694 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  5. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  6. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  7. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  9. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  10. 2023A Comprehensive Overview of Large Language Modelscited by 639
  11. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  12. 2023Active Retrieval Augmented Generationcited by 506
38 more
  1. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  2. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  3. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  4. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  5. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  6. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  7. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  8. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  9. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  10. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  11. 2024LightRAG: Simple and Fast Retrieval-Augmented Generationcited by 310
  12. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 275
  13. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  14. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  15. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  16. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  17. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  18. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  19. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  20. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 200
  21. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  22. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  23. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  24. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  25. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  26. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  27. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  28. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  29. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 150
  30. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
  31. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 142
  32. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  33. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 125
  34. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  35. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 118
  36. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 114
  37. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  38. 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110

50 most cited of 694 All 694 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024FlashBack: Efficient Retrieval-Augmented Language Modeling for Fast Inference
  2. 2025Do Retrieval-Augmented Language Models Adapt to Varying User Needs?
  3. 2023RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models
  4. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models
  5. 2024R-Eval: A Unified Toolkit for Evaluating Domain Knowledge of Retrieval Augmented Large Language Models
  6. 2024R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generation
  7. 2024Improving Retrieval Augmented Language Model with Self-Reasoning
  8. 2024More Room for Language: Investigating the Effect of Retrieval on Language Models