mlsanityData to 7 Oct 2026

Retrieval-Enhanced Machine Learning: Synthesis and Opportunities

arXiv 2407.12982 ↗First posted 17 Jul 2024cs.LG cs.CL cs.IRcited by 11

Not on the map. Why?

Abstract (arXiv)

In the field of language modeling, models augmented with retrieval components have emerged as a promising solution to address several challenges faced in the natural language processing (NLP) field, including knowledge grounding, interpretability, and scalability. Despite the primary focus on NLP, we posit that the paradigm of retrieval-enhancement can be extended to a broader spectrum of machine learning (ML) such as computer vision, time series prediction, and computational biology. Therefore, this work introduces a formal framework of this paradigm, Retrieval-Enhanced Machine Learning (REML), by synthesizing the literature in various domains in ML with consistent notations which is missing from the current literature. Also, we found that while a number of studies employ retrieval components to augment their models, there is a lack of integration with foundational Information Retrieval (IR) research. We bridge this gap between the seminal IR research and contemporary REML studies by investigating each component that comprises the REML framework. Ultimately, the goal of this work is to equip researchers across various disciplines with a comprehensive, formally structured framework of retrieval-enhanced models, thereby fostering interdisciplinary future research.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 191 of its 258 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

191 of 258 references.

References · 191

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  12. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
38 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2020Extracting Training Data from Large Language Modelscited by 2,144
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2021ViViT: A Video Vision Transformercited by 1,610
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  10. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  13. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  14. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  15. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  16. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  17. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  18. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  19. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  20. 2015Pointer Networkscited by 882
  21. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  22. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  23. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  24. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  25. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  26. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  27. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  28. 2022A Survey on In-context Learningcited by 717
  29. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  30. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  31. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  32. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  33. 2014Neural Turing Machinescited by 630
  34. 2019Passage Re-ranking with BERTcited by 629
  35. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  36. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  37. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  38. 2019ELI5: Long Form Question Answeringcited by 567

50 most cited of 191 All 191 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 2; 2026: 3.

11 papers.

  1. 2024Long Context RAG Performance of Large Language Modelscited by 23
  2. 2024Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Modelscited by 20
  3. 2024Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generationcited by 15
  4. 2024Learning to Rank for Multiple Retrieval-Augmented Models through Iterative Utility Maximizationcited by 14
  5. 2024Accelerating Retrieval-Augmented Generationcited by 14
  6. 2025CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Trainingcited by 2
  7. 2026Who Benefits from RAG? The Role of Exposure, Utility and Attribution Biascited by 1
  8. 2026Multi-Agent Transactive Memorycited by 1
  9. 2024LML-DAP: Language Model Learning a Dataset for Data-Augmented Predictioncited by 0
  10. 2025TeamCMU at Touch'e: Adversarial Co-Evolution for Advertisement Integration and Detection in Conversational Searchcited by 0
  11. 2026Improving User Privacy in Personalized Generation: Client-Side Retrieval-Augmented Modification of Server-Side Generated Speculationscited by 0