# Retrieval-Enhanced Machine Learning

arXiv: https://arxiv.org/abs/2205.01230 · HTML: https://mlsanity.com/paper/2205.01230/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 80 references that are in the map.
- First posted on arXiv 2 May 2022.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [Transformer Memory as a Differentiable Search Index](https://mlsanity.com/paper/2202.06991/) (2022)
- [Long-Term Feature Banks for Detailed Video Understanding](https://mlsanity.com/paper/1812.05038/) (2018)
- [Internet-Augmented Dialogue Generation](https://mlsanity.com/paper/2107.07566/) (2021)
- [Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2010.00710/) (2020)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [Deeper Text Understanding for IR with Contextual Neural Language Modeling](https://mlsanity.com/paper/1905.09217/) (2019)

## Built on by (most cited first, 30 of 47)

- [Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation](https://mlsanity.com/paper/2404.05970/) (2024)
- [Evaluating Retrieval Quality in Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.13781/) (2024)
- [FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2209.14290/) (2022)
- [Leveraging Large Language Models in Conversational Recommender Systems](https://mlsanity.com/paper/2305.07961/) (2023)
- [Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Community](https://mlsanity.com/paper/2307.09751/) (2023)
- [Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility Maximization](https://mlsanity.com/paper/2405.02816/) (2024)
- [Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.00175/) (2024)
- [Adaptive Computation with Elastic Input Sequence](https://mlsanity.com/paper/2301.13195/) (2023)
- [Are Large Language Models Good at Utility Judgments?](https://mlsanity.com/paper/2403.19216/) (2024)
- [Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Models](https://mlsanity.com/paper/2409.09510/) (2024)
- [Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding](https://mlsanity.com/paper/2404.14600/) (2024)
- [You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LM](https://mlsanity.com/paper/2210.15859/) (2022)
- [Improving Retrieval-Augmented Large Language Models via Data Importance Learning](https://mlsanity.com/paper/2307.03027/) (2023)
- [Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.11598/) (2024)
- [A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answering](https://mlsanity.com/paper/2304.13649/) (2023)
- [Learning to Rank for Multiple Retrieval-Augmented Models through Iterative Utility Maximization](https://mlsanity.com/paper/2410.09942/) (2024)
- [Accelerating Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.15246/) (2024)
- [Cross-modal Retrieval for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2401.05736/) (2024)
- [Retrieval-Enhanced Machine Learning: Synthesis and Opportunities](https://mlsanity.com/paper/2407.12982/) (2024)
- [Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling](https://mlsanity.com/paper/2207.03030/) (2022)
- [KNN-LM Does Not Improve Open-ended Text Generation](https://mlsanity.com/paper/2305.14625/) (2023)
- [GrepSeek: Training Search Agents for Direct Corpus Interaction](https://mlsanity.com/paper/2605.29307/) (2026)
- [Query Understanding in the Age of Large Language Models](https://mlsanity.com/paper/2306.16004/) (2023)
- [RRAML: Reinforced Retrieval Augmented Machine Learning](https://mlsanity.com/paper/2307.12798/) (2023)
- [Retrieval-Augmented Mining of Temporal Logic Specifications from Data](https://mlsanity.com/paper/2405.14355/) (2024)
- [LTRR: Learning To Rank Retrievers for LLMs](https://mlsanity.com/paper/2506.13743/) (2025)
- [RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.06056/) (2025)
- [Multivariate Representation Learning for Information Retrieval](https://mlsanity.com/paper/2304.14522/) (2023)
- [Multi-Class Quantum Convolutional Neural Networks](https://mlsanity.com/paper/2404.12741/) (2024)
- [Retrieval-Augmented Feature Generation for Domain-Specific Classification](https://mlsanity.com/paper/2406.11177/) (2024)

## Abstract (arXiv)

Although information access systems have long supported people in accomplishing a wide range of tasks, we propose broadening the scope of users of information access systems to include task-driven machines, such as machine learning models. In this way, the core principles of indexing, representation, retrieval, and ranking can be applied and extended to substantially improve model generalization, scalability, robustness, and interpretability. We describe a generic retrieval-enhanced machine learning (REML) framework, which includes a number of existing models as special cases. REML challenges information retrieval conventions, presenting opportunities for novel advances in core areas, including optimization. The REML research agenda lays a foundation for a new style of information access research and paves a path towards advancing machine learning and artificial intelligence.