mlsanityData to 14 Oct 2026

Magic Mushroom: A Customizable Benchmark for Fine-grained Analysis of Retrieval Noise Erosion in RAG Systems

arXiv 2506.03901 ↗First posted 4 Jun 2025cs.CLcited by 1

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) by incorporating external retrieved information, mitigating issues such as hallucination and outdated knowledge. However, RAG systems are highly sensitive to retrieval noise prevalent in real-world scenarios. Existing benchmarks fail to emulate the complex and heterogeneous noise distributions encountered in real-world retrieval environments, undermining reliable robustness assessment. In this paper, we define four categories of retrieval noise based on linguistic properties and noise characteristics, aiming to reflect the heterogeneity of noise in real-world scenarios. Building on this, we introduce Magic Mushroom, a benchmark for replicating "magic mushroom" noise: contexts that appear relevant on the surface but covertly mislead RAG systems. Magic Mushroom comprises 7,468 single-hop and 3,925 multi-hop question-answer pairs. More importantly, Magic Mushroom enables researchers to flexibly configure combinations of retrieval noise according to specific research objectives or application scenarios, allowing for highly controlled evaluation setups. We evaluate LLM generators of varying parameter scales and classic RAG denoising strategies under diverse noise distributions to investigate their performance dynamics during progressive noise encroachment. Our analysis reveals that both generators and denoising strategies have significant room for improvement and exhibit extreme sensitivity to noise distributions. Magic Mushroom emerges as a promising tool for evaluating and advancing noise-robust RAG systems, accelerating their widespread deployment in real-world applications. The Magic Mushroom benchmark is available at https://drive.google.com/file/d/1aP5kyPuk4L-L_uoI6T9UhxuTyt8oMqjT/view?usp=sharing.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 38 of its 44 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

38 of 44 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defects30 Jan
  2. 2024Pandora's Box or Aladdin's Lamp: A Comprehensive Analysis Revealing the Role of RAG Noise in Large Language Models24 Aug
  3. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering19 Jul
  4. 2024WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia19 Jun
  5. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training31 May
  6. 2024RAGGED: Towards Informed Design of Scalable and Stable RAG Systems14 Mar
  7. 2023"Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation18 Dec
  8. 2023Learning to Filter Context for Retrieval-Augmented Generation14 Nov
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection17 Oct
  10. 2023Benchmarking Large Language Models in Retrieval-Augmented Generation4 Sep

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
16 more
  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  2. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  3. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  4. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  5. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  6. 2024Corrective Retrieval Augmented Generationcited by 197
  7. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 184
  8. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  9. 2023Self-Knowledge Guided Retrieval Augmentation for Large Language Modelscited by 69
  10. 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
  11. 2024RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewardscited by 33
  12. 2024Improving Retrieval Augmented Language Model with Self-Reasoningcited by 28
  13. 2021RetrievalSum: A Retrieval Enhanced Framework for Abstractive Summarizationcited by 19
  14. 2024Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generationcited by 19
  15. 2024Hierarchical Indexing for Retrieval-Augmented Opinion Summarizationcited by 7
  16. 2024Targeted Augmentation for Low-Resource Event Extractioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  2. 2025Finetune-RAG: Fine-Tuning Language Models to Resist Hallucination in Retrieval-Augmented Generation
  3. 2026Tackling the Inherent Difficulty of Noise Filtering in RAG
  4. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
  5. 2025RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition
  6. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity
  7. 2025Evaluating the Retrieval Robustness of Large Language Models
  8. 2025PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation