mlsanityData to 14 Oct 2026

Retrieval-Guided Fine-Tuning as Noisy Estimation: Risk bounds and Architectural Analysis

arXiv 2609.14485 ↗First posted 13 Sep 2026cs.LG math.ST stat.THcited by 0

Not on the map. Why?

Abstract (arXiv)

Retrieval-Guided Fine-Tuning (RAG-FT) incorporates retrieved data directly into the training objective, but the statistical consequences of noisy retrieval during training remain theoretically undercharacterized. We study this question by modeling RAG-FT as an estimation problem in a multi-task linear regression framework, using an OLS proxy for single-layer linear self-attention to obtain finite-sample risk bounds. Under homoscedastic retrieval noise, we show that retrieval failure decays exponentially with task separation relative to noise, and derive explicit finite-sample conditions under which RAG-FT achieves lower risk than both target-only and full-corpus training. We then introduce a Distance-Proportional Noise (DPN) model, in which retrieval quality degrades with rank, and compare two estimators under the same retrieval process: the OLS proxy and the literal, uniform-weight forward pass of linear self-attention. We prove that the attention estimator's bias diverges as Theta(n2q) even under exact retrieval, while OLS risk remains Theta(d/n) for every noise exponent q>0. These results locate the instability not in noisy retrieval itself, but in the fixed, unweighted aggregation of the literal LSA forward pass, which reweighting by reliability empirically removes. We validate the predicted rate separation through direct simulation of the DPN model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 26 references that are in the map. First posted on arXiv 13 Sep 2026.

Builds on

21 of 26 references.

References · 21

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  5. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  6. 2022Transformers learn in-context by gradient descentcited by 732
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  9. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  10. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  11. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  12. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
9 more
  1. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
  2. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 256
  3. 2015The Benefit of Multitask Representation Learningcited by 195
  4. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  5. 2020Provable Meta-Learning of Linear Representationscited by 161
  6. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  7. 2023A Closer Look at In-Context Learning under Distribution Shiftscited by 25
  8. 2025Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Boundscited by 4
  9. 2025Dual Debiasing for Noisy In-Context Learning for Text Generationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Tackling the Inherent Difficulty of Noise Filtering in RAG
  2. 2025Optimizing What Matters: AUC-Driven Learning for Robust Neural Retrieval
  3. 2026The Powerless Noise: How Experimental Settings Shape the Reported Power of Noise
  4. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training
  5. 2025RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defects
  6. 2026Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generation
  7. 2025BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generation
  8. 2026When Does Dense Retrieval Need Asymmetric Geometry? A Bias-Variance Theory of Shared and Dual Projections