mlsanityData to 14 Oct 2026

Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise

arXiv 2305.01579 ↗First posted 2 May 2023cs.CL cs.AIcited by 54

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Most existing retrieval-augmented language models (LMs) assume a naive dichotomy within a retrieved document set: query-relevance and irrelevance. Our work investigates a more challenging scenario in which even the "relevant" documents may contain misleading or incorrect information, causing conflict among the retrieved documents and thereby negatively influencing model decisions as noise. We observe that existing LMs are highly brittle to the presence of conflicting information in both the fine-tuning and in-context few-shot learning scenarios. We propose approaches for handling knowledge conflicts among retrieved documents by explicitly fine-tuning a discriminator or prompting GPT-3.5 to elicit its discriminative capability. Our empirical results on open-domain QA show that these approaches significantly enhance model robustness. We also provide our findings on incorporating the fine-tuned discriminator's decision into the in-context learning process, proposing a way to exploit the benefits of two disparate learning schemes. Alongside our findings, we provide MacNoise, a machine-generated, conflict-induced dataset to further encourage research in this direction.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 25 of its 44 references that are in the map. First posted on arXiv 2 May 2023.

Builds on

25 of 44 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
13 more
  1. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  2. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  3. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  4. 2022Prompting GPT-3 To Be Reliablecited by 260
  5. 2020Selective Question Answering under Domain Shiftcited by 238
  6. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 200
  7. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  8. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 144
  9. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 96
  10. 2021Knowing More About Questions Can Help: Improving Calibration in Question Answeringcited by 47
  11. 2020FiD-Ex: Improving Sequence-to-Sequence Models for Extractive Rationale Generationcited by 21
  12. 2022Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Expertscited by 10
  13. 2021Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrievalcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 16; 2026: 24.

54 papers.

Sort:

By area:

  1. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  2. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 63
  3. 2023InGram: Inductive Knowledge Graph Embedding via Relation Graphscited by 52
  4. 2024Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Modelscited by 43
  5. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  6. 2024RevPRAG: Revealing Poisoning Attacks in Retrieval-Augmented Generation through LLM Activation Analysiscited by 33
  7. 2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attackscited by 24
  8. 2024Controllable Context Sensitivity and the Knob Behind Itcited by 23
  9. 2024CrAM: Credibility-Aware Attention Modification in LLMs for Combating Misinformation in RAGcited by 22
  10. 2024Retrieval-Augmented Generation with Estimation of Source Reliabilitycited by 19
  11. 2024Open Domain Question Answering with Conflicting Contextscited by 18
  12. 2024Analysing the Residual Stream of Language Models Under Knowledge Conflictscited by 17
38 more
  1. 2025Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMscited by 16
  2. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15
  3. 2025Shifting from Ranking to Set Selection for Retrieval Augmented Generationcited by 15
  4. 2025Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactionscited by 13
  5. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  6. 2026Whose Facts Win? LLM Source Preferences under Knowledge Conflictscited by 12
  7. 2025Rescuing the Unpoisoned: Efficient Defense against Knowledge Corruption Attacks on RAG Systemscited by 9
  8. 2025SUGAR: Leveraging Contextual Confidence for Smarter Retrievalcited by 7
  9. 2024Enhancing Robustness of Retrieval-Augmented Language Models with In-Context Learningcited by 6
  10. 2025How does Misinformation Affect Large Language Model Behaviors and Preferences?cited by 5
  11. 2025After Retrieval, Before Generation: Enhancing the Trustworthiness of Large Language Models in Retrieval-Augmented Generationcited by 4
  12. 2025Exploring the Impact of Instruction-Tuning on LLM's Susceptibility to Misinformationcited by 4
  13. 2025Attacks by Content: Automated Fact-checking is an AI Security Issuecited by 4
  14. 2026Is Deep Research Reliable? Misleading Knowledge Induces False Conclusionscited by 4
  15. 2025When to Trust Context: Self-Reflective Debates for Context Reliabilitycited by 3
  16. 2026Addressing Corpus Knowledge Poisoning Attacks on RAG Using Sparse Attentioncited by 3
  17. 2026MemGuard: Preventing Memory Contamination in Long-Term Memory-Augmented Large Language Modelscited by 3
  18. 2025Quantifying Memorization and Parametric Response Rates in Retrieval-Augmented Vision-Language Modelscited by 2
  19. 2025SegSub: Evaluating Robustness to Knowledge Conflicts and Hallucinations in Vision-Language Modelscited by 2
  20. 2025Conflicts in Texts: Data, Implications and Challengescited by 2
  21. 2026Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directionscited by 2
  22. 2026RAGuard: A Layered Defense Framework for Retrieval-Augmented Generation Systems Against Data Poisoningcited by 2
  23. 2026Mitigating Hallucination in Financial Retrieval-Augmented Generation via Fine-Grained Knowledge Verificationcited by 1
  24. 2026The Impact of Ideological Discourses in RAG: A Case Study with COVID-19 Treatmentscited by 1
  25. 2026Evaluating Evidence Grounding Under User Pressure in Instruction-Tuned Language Modelscited by 1
  26. 2026Navigating Unreliable Parametric and Contextual Knowledge: Explicit Knowledge Conflict Resolution for LLM Inferencecited by 1
  27. 2026SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematicscited by 1
  28. 2026Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluationcited by 1
  29. 2026HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Documentcited by 1
  30. 2026Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QAcited by 1
  31. 2026Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generationcited by 1
  32. 2025Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncodercited by 0
  33. 2025On Finding Inconsistencies in Documentscited by 0
  34. 2026Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generationcited by 0
  35. 2026Verify as You Go: An LLM-Powered Browser Extension for Fake News Detectioncited by 0
  36. 2026MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studiescited by 0
  37. 2026ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidencecited by 0
  38. 2026MEDIAREF: A Public Knowledge Store for Media Background Checkscited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Toward Robust RALMs: Revealing the Impact of Imperfect Retrieval on Retrieval-Augmented Language Models
  2. 2025Accommodate Knowledge Conflicts in Retrieval-augmented LLMs: Towards Robust Response Generation in the Wild
  3. 2024Assessing "Implicit" Retrieval Robustness of Large Language Models
  4. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models
  5. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence
  6. 2025Retrieval-Augmented Generation with Conflicting Evidence
  7. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training
  8. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context