mlsanityData to 14 Oct 2026

Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds

arXiv 2506.03100 ↗First posted 3 Jun 2025cs.LG cs.AI cs.CLcited by 4

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-augmented generation (RAG) has seen many empirical successes in recent years by aiding the LLM with external knowledge. However, its theoretical aspect has remained mostly unexplored. In this paper, we propose the first finite-sample generalization bound for RAG in in-context linear regression and derive an exact bias-variance tradeoff. Our framework views the retrieved texts as query-dependent noisy in-context examples and recovers the classical in-context learning (ICL) and standard RAG as the limit cases. Our analysis suggests that an intrinsic ceiling on generalization error exists on RAG as opposed to the ICL. Furthermore, our framework is able to model retrieval both from the training data and from external corpora by introducing uniform and non-uniform RAG noise. In line with our theory, we show the sample efficiency of ICL and RAG empirically with experiments on common QA benchmarks, such as Natural Questions and TriviaQA.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 39 of its 50 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

39 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025More Documents, Same Length: Isolating the Challenge of Multiple Documents in RAG6 Mar
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Survey29 Feb

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  6. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  8. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  10. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  12. 2023In-Context Retrieval-Augmented Language Modelscited by 702
25 more
  1. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  2. 2021MetaICL: Learning to Learn In Contextcited by 454
  3. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  4. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  5. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  6. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  7. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  8. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 151
  9. 2023Compositional Exemplars for In-context Learningcited by 144
  10. 2023Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sourcescited by 122
  11. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  12. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  13. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 78
  14. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 75
  15. 2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70
  16. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 70
  17. 2022Self-Prompting Large Language Models for Zero-Shot Open-Domain QAcited by 39
  18. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  19. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Modelscited by 37
  20. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 33
  21. 2020Is Retriever Merely an Approximator of Reader?cited by 30
  22. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  23. 2025Test-Time Training Provably Improves Transformers as In-context Learnerscited by 16
  24. 2024Exploring the Robustness of In-Context Learning with Noisy Labelscited by 14
  25. 2024NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systemscited by 6
  2. 2026Retrieval-Guided Fine-Tuning as Noisy Estimation: Risk bounds and Architectural Analysiscited by 0
  3. 2026Retrieval-Based In-Context Learning: A Domain Adaptation Frameworkcited by 0
  4. 2026SpecForge: Recursive Self-Improvement for Coding Agents via Behavioral Specificationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective
  2. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  3. 2025Dynamic Context Selection for Retrieval-Augmented Generation: Mitigating Distractors and Positional Bias
  4. 2024Towards Optimizing a Retrieval Augmented Generation using Large Language Model on Academic Data
  5. 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation
  6. 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practices
  7. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
  8. 2024R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generation