# The Distracting Effect: Understanding Irrelevant Passages in RAG

arXiv: https://arxiv.org/abs/2505.06914 · HTML: https://mlsanity.com/paper/2505.06914/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 11 May 2025.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 19 of 19)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [RA-DIT: Retrieval-Augmented Dual Instruction Tuning](https://mlsanity.com/paper/2310.01352/) (2023)
- [Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2311.09210/) (2023)
- [Large Language Models with Controllable Working Memory](https://mlsanity.com/paper/2211.05110/) (2022)
- [Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG](https://mlsanity.com/paper/2410.05983/) (2024)
- [InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales](https://mlsanity.com/paper/2406.13629/) (2024)
- [NV-Retriever: Improving text embedding models with effective hard-negative mining](https://mlsanity.com/paper/2407.15831/) (2024)
- [Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models](https://mlsanity.com/paper/2409.11136/) (2024)
- [LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements](https://mlsanity.com/paper/2404.06283/) (2024)

## Built on by (most cited first, 30 of 33)

- [Do RAG Systems Really Suffer From Positional Bias?](https://mlsanity.com/paper/2505.15561/) (2025)
- [What Papers Don't Tell You: Recovering Tacit Knowledge for Automated Paper Reproduction](https://mlsanity.com/paper/2603.01801/) (2026)
- [Prompting Large Language Models with Partial Knowledge for Answering Questions with Unseen Entities](https://mlsanity.com/paper/2508.01290/) (2025)
- [Redefining Retrieval Evaluation in the Era of LLMs](https://mlsanity.com/paper/2510.21440/) (2025)
- [Dynamic Context Selection for Retrieval-Augmented Generation: Mitigating Distractors and Positional Bias](https://mlsanity.com/paper/2512.14313/) (2025)
- [Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering](https://mlsanity.com/paper/2603.26567/) (2026)
- [Memory Retrieval for Changing Preferences](https://mlsanity.com/paper/2606.02976/) (2026)
- [Large Language Models Lack Temporal Awareness of Medical Knowledge](https://mlsanity.com/paper/2605.13045/) (2026)
- [MeVe: A Modular System for Memory Verification and Effective Context Control in Language Models](https://mlsanity.com/paper/2509.01514/) (2025)
- [Relevance to Utility: Process-Supervised Rewrite for RAG](https://mlsanity.com/paper/2509.15577/) (2025)
- [How important is Recall for Measuring Retrieval Quality?](https://mlsanity.com/paper/2512.20854/) (2025)
- [Predicting the Benefit of Retrieval Augmentation in Open-Domain Question Answering](https://mlsanity.com/paper/2604.07985/) (2026)
- [Purifying Multimodal Retrieval: Fragment-Level Evidence Selection for RAG](https://mlsanity.com/paper/2604.27600/) (2026)
- [LLM-Oriented Information Retrieval: A Denoising-First Perspective](https://mlsanity.com/paper/2605.00505/) (2026)
- [Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG](https://mlsanity.com/paper/2605.27105/) (2026)
- [On the impact of retrieved content representations in RAG Pipelines](https://mlsanity.com/paper/2605.30790/) (2026)
- [The Powerless Noise: How Experimental Settings Shape the Reported Power of Noise](https://mlsanity.com/paper/2607.03615/) (2026)
- [Predicting Partial Answer Quality and Utility in Agentic Retrieval-Augmented Generation](https://mlsanity.com/paper/2609.16453/) (2026)
- [Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding](https://mlsanity.com/paper/2509.21865/) (2025)
- [ReFilter: Improving Robustness of Retrieval-Augmented Generation via Gated Filter](https://mlsanity.com/paper/2602.12709/) (2026)
- [Mitigating LLM Hallucinations through Domain-Grounded Tiered Retrieval](https://mlsanity.com/paper/2603.17872/) (2026)
- [Hypothesis-Conditioned Query Rewriting for Decision-Useful Retrieval](https://mlsanity.com/paper/2603.19008/) (2026)
- [Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articles](https://mlsanity.com/paper/2603.21193/) (2026)
- [Natural Language Query to Configuration for Retrieval Agents](https://mlsanity.com/paper/2605.27361/) (2026)
- [Memory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agents](https://mlsanity.com/paper/2606.25361/) (2026)
- [Polistemics: Evaluating LLMs as Information Mediators in Politics & Elections](https://mlsanity.com/paper/2607.25953/) (2026)
- [GuidedRAG: Semantic Steering of Retrieval-Augmented Generation](https://mlsanity.com/paper/2607.26071/) (2026)
- [When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models](https://mlsanity.com/paper/2608.19208/) (2026)
- [Research Design Tracking and Assessment for the Social Sciences](https://mlsanity.com/paper/2608.27049/) (2026)
- [Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation](https://mlsanity.com/paper/2608.30996/) (2026)

## Abstract (arXiv)

A well-known issue with Retrieval Augmented Generation (RAG) is that retrieved passages that are irrelevant to the query sometimes distract the answer-generating LLM, causing it to provide an incorrect response. In this paper, we shed light on this core issue and formulate the distracting effect of a passage w.r.t. a query (and an LLM). We provide a quantifiable measure of the distracting effect of a passage and demonstrate its robustness across LLMs. Our research introduces novel methods for identifying and using hard distracting passages to improve RAG systems. By fine-tuning LLMs with these carefully selected distracting passages, we achieve up to a 7.5% increase in answering accuracy compared to counterparts fine-tuned on conventional RAG datasets. Our contribution is two-fold: first, we move beyond the simple binary classification of irrelevant passages as either completely unrelated vs. distracting, and second, we develop and analyze multiple methods for finding hard distracting passages. To our knowledge, no other research has provided such a comprehensive framework for identifying and utilizing hard distracting passages.