mlsanityData to 7 Oct 2026

A Statistical Framework for Data-dependent Retrieval-Augmented Models

arXiv 2408.15399 ↗First posted 27 Aug 2024cs.LG cs.AI cs.CLcited by 2

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Modern ML systems increasingly augment input instances with additional relevant information to enhance final prediction. Despite growing interest in such retrieval-augmented models, their fundamental properties and training are not well understood. We propose a statistical framework to study such models with two components: 1) a em retriever to identify the relevant information out of a large corpus via a data-dependent metric; and 2) a em predictor that consumes the input instances along with the retrieved information to make the final predictions. We present a principled method for end-to-end training of both components and draw connections with various training approaches in the literature. Furthermore, we establish excess risk bounds for retrieval-augmented models while delineating the contributions of both retriever and predictor towards the model performance. We validate the utility of our proposed training methods along with the key takeaways from our statistical analysis on open domain question answering task where retrieval augmentation is important.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 50 of its 80 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

50 of 80 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuning2 Oct
  2. 2022Variational Open-Domain Question Answering23 Sep
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
33 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  7. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  8. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  9. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  10. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  11. 2016Error bounds for approximations with deep ReLU networkscited by 567
  12. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  13. 2015Importance Weighted Autoencoderscited by 454
  14. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  15. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  16. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  17. 2022Test-Time Training with Masked Autoencoderscited by 208
  18. 2021Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptationcited by 176
  19. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  20. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  21. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 123
  22. 2021Case-based Reasoning for Natural Language Queries over Knowledge Basescited by 107
  23. 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
  24. 2021Oops I Took A Gradient: Scalable Sampling for Discrete Distributionscited by 92
  25. 2015How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?cited by 76
  26. 2022Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Datacited by 61
  27. 2022Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spacescited by 50
  28. 2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
  29. 2021Hindsight: Posterior-guided training of retrievers for improved open-ended generationcited by 38
  30. 2023Improving Image Recognition by Retrieving from Web-Scale Image-Text Datacited by 20
  31. 2022Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsingcited by 15
  32. 2020Optimal Approximation -- Smoothness Tradeoffs for Soft-Max Functionscited by 10
  33. 2023Machine Reading Comprehension using Case-based Reasoningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025RAISE: Enhancing Scientific Reasoning in LLMs via Step-by-Step Retrievalcited by 1
  2. 2026ARMOR: Adaptive Retriever Optimization for Low-Resource Telecom Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving Retrieval-Augmented Large Language Models via Data Importance Learning
  2. 2023BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer
  3. 2024Unraveling and Mitigating Retriever Inconsistencies in Retrieval-Augmented Large Language Models
  4. 2026Predicting the Benefit of Retrieval Augmentation in Open-Domain Question Answering
  5. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context
  6. 2024Reliable, Adaptable, and Attributable Language Models with Retrieval
  7. 2026RPDR: A Round-trip Prediction-Based Data Augmentation Framework for Long-Tail Question Answering
  8. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval