mlsanityData to 7 Oct 2026

REPLUG: Retrieval-Augmented Black-Box Language Models

arXiv 2301.12652 ↗First posted 30 Jan 2023cs.CLcited by 786

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing retrieval and language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improves the performance of GPT-3 (175B) on language modeling by 6.3%, as well as the performance of Codex on five-shot MMLU by 5.1%.

Key facts. Cited by 786 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 38 of its 46 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

38 of 46 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP28 Dec
  2. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  3. 2022kNN-Prompt: Nearest Neighbor Zero-Shot Inference27 May
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2017Billion-scale similarity search with GPUscited by 2,477
21 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  7. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  8. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  9. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  10. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  11. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  12. 2022Prompting GPT-3 To Be Reliablecited by 258
  13. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  14. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  15. 2022Training Language Models with Memory Augmentationcited by 112
  16. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  17. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  18. 2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
  19. 2022Nonparametric Masked Language Modelingcited by 39
  20. 2021R2-D2: A Modular Baseline for Open-Domain Question Answeringcited by 37
  21. 2021Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learningcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 145; 2024: 288; 2025: 220; 2026: 133.

786 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  4. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  5. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  8. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  9. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  12. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
38 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 639
  2. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  3. 2023Active Retrieval Augmented Generationcited by 506
  4. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  5. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  6. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  7. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  8. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  9. 2023Augmented Language Models: a Surveycited by 398
  10. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  11. 2024The Faiss librarycited by 370
  12. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  13. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  14. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  15. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  16. 2023Adapting Language Models to Compress Contextscited by 350
  17. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
  18. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  19. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  20. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  21. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  22. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  23. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  24. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  25. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  26. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  27. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  28. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  29. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  30. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  31. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  32. 2024Generative Representational Instruction Tuningcited by 227
  33. 2024Tool Learning with Large Language Models: A Surveycited by 221
  34. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  35. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  36. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  37. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  38. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201

50 most cited of 786 All 786 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer
  2. 2024MemLong: Memory-Augmented Retrieval for Long Text Modeling
  3. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study
  4. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval
  5. 2024More Room for Language: Investigating the Effect of Retrieval on Language Models
  6. 2025Do Retrieval-Augmented Language Models Adapt to Varying User Needs?
  7. 2023On the Generalization Ability of Retrieval-Enhanced Transformers
  8. 2024Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Models