mlsanityData to 14 Oct 2026

Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMs

arXiv 2505.17762 ↗First posted 23 May 2025cs.CL cs.IRcited by 16

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Large Language Models (LLMs) augmented with retrieval mechanisms have demonstrated significant potential in fact-checking tasks by integrating external knowledge. However, their reliability decreases when confronted with conflicting evidence from sources of varying credibility. This paper presents the first systematic evaluation of Retrieval-Augmented Generation (RAG) models for fact-checking in the presence of conflicting evidence. To support this study, we introduce CONFACT (Conflicting Evidence for Fact-Checking) (Dataset available at https://github.com/zoeyyes/CONFACT), a novel dataset comprising questions paired with conflicting information from various sources. Extensive experiments reveal critical vulnerabilities in state-of-the-art RAG methods, particularly in resolving conflicts stemming from differences in media source credibility. To address these challenges, we investigate strategies to integrate media background information into both the retrieval and generation stages. Our results show that effectively incorporating source credibility significantly enhances the ability of RAG models to resolve conflicting evidence and improve fact-checking performance.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 39 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

31 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Generating Media Background Checks for Automated Source Critical Reasoning1 Sep
  2. 2024Reinforcement Retrieval Leveraging Fine-grained Feedback for Fact Checking News Claims with Black-Box LLM26 Apr
  3. 2023Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise2 May

Other references · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2024Qwen2 Technical Reportcited by 2,120
  8. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  9. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  10. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  11. 2021A Survey on Automated Fact-Checkingcited by 378
  12. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 363
16 more
  1. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  2. 2023Fact-Checking Complex Claims with Program-Guided Reasoningcited by 146
  3. 2021Automated Fact-Checking for Assisting Human Fact-Checkerscited by 144
  4. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  5. 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110
  6. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  7. 2018Predicting Factuality of Reporting and Bias of News Media Sourcescited by 71
  8. 2023A Survey of Large Language Models Attributioncited by 71
  9. 2020What Was Written vs. Who Read It: News Media Profiling Using Text Analysis and Social Media Contextcited by 42
  10. 2021Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLPcited by 31
  11. 2023QACHECK: A Demonstration System for Question-Guided Multi-Hop Fact-Checkingcited by 26
  12. 2024AmbigDocs: Reasoning across Documents on Different Entities under the Same Namecited by 21
  13. 2019Tanbih: Get To Know What You Are Readingcited by 16
  14. 2017People on Media: Jointly Identifying Credible News and Trustworthy Citizen Journalists in Online Communitiescited by 10
  15. 2024MegaFake: A Theory-Driven Dataset of Fake News Generated by Large Language Modelscited by 7
  16. 2023Trusted Source Alignment in Large Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 11.

16 papers.

Sort:

By area:

  1. 2025Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PDcited by 13
  2. 2025Retrieval- and Argumentation-Enhanced Multi-Agent LLMs for Judgmental Forecasting (Extended Version with Supplementary Material)cited by 8
  3. 2025Debating Truth: Debate-driven Claim Verification with Multiple Large Language Model Agentscited by 6
  4. 2026Courtroom-Style Multi-Agent Debate with Progressive RAG and Role-Switching for Controversial Claim Verificationcited by 4
  5. 2026ConflictRAG: Detecting and Resolving Knowledge Conflicts in Retrieval Augmented Generationcited by 4
  6. 2025HaluNet: Learning Hallucination Risk from Internal Signals in LLM Question Answeringcited by 2
  7. 2026The Automatic Verification of Image-Text Claims (AVerImaTeC) Shared Taskcited by 2
  8. 2026Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Predictioncited by 1
  9. 2025Towards Real-Time Fake News Detection under Evidence Scarcitycited by 0
  10. 2026The Alignment Bottleneck in Decomposition-Based Claim Verificationcited by 0
  11. 2026ArbGraph: Conflict-Aware Evidence Arbitration for Reliable Long-Form Retrieval-Augmented Generationcited by 0
  12. 2026When Specifications Conflict: A Symmetry-Based Framework for Measuring LLM Preferencescited by 0
4 more
  1. 2026When Context Misleads: Intent-Guided Decoding for Robust Retrieval-Augmented Generationcited by 0
  2. 2026CoVer: Conflict-Aware Claim Verificationcited by 0
  3. 2026Record Grouping Controls Evidence Weight in Language Modelscited by 0
  4. 2026Multi-Channel Mitigation of Source-Trust Shortcuts in Fact-Checking RL Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals
  2. 2024Face the Facts! Evaluating RAG-based Pipelines for Professional Fact-Checking
  3. 2025Fact-checking AI-generated news reports: Can LLMs catch their own lies?
  4. 2026MEDIAREF: A Public Knowledge Store for Media Background Checks
  5. 2026Evaluating RAG Reliability under Clean, Misleading, and Mixed Retrieval
  6. 2025Rethinking All Evidence: Enhancing Trustworthy Retrieval-Augmented Generation via Conflict-Driven Summarization
  7. 2024RAGAR, Your Falsehood Radar: RAG-Augmented Reasoning for Political Fact-Checking using Multimodal Large Language Models
  8. 2025From Facts to Conclusions : Integrating Deductive Reasoning in Retrieval-Augmented LLMs