# Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMs

arXiv: https://arxiv.org/abs/2505.17762 · HTML: https://mlsanity.com/paper/2505.17762/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 30 of 31)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Fact-Checking Complex Claims with Program-Guided Reasoning](https://mlsanity.com/paper/2305.12744/) (2023)
- [Automated Fact-Checking for Assisting Human Fact-Checkers](https://mlsanity.com/paper/2103.07769/) (2021)
- [AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web](https://mlsanity.com/paper/2305.13117/) (2023)
- [Complex Claim Verification with Evidence Retrieved in the Wild](https://mlsanity.com/paper/2305.11859/) (2023)
- [Searching for Best Practices in Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.01219/) (2024)
- [Predicting Factuality of Reporting and Bias of News Media Sources](https://mlsanity.com/paper/1810.01765/) (2018)
- [A Survey of Large Language Models Attribution](https://mlsanity.com/paper/2311.03731/) (2023)
- [Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise](https://mlsanity.com/paper/2305.01579/) (2023)
- [What Was Written vs. Who Read It: News Media Profiling Using Text Analysis and Social Media Context](https://mlsanity.com/paper/2005.04518/) (2020)
- [Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP](https://mlsanity.com/paper/2106.06830/) (2021)
- [QACHECK: A Demonstration System for Question-Guided Multi-Hop Fact-Checking](https://mlsanity.com/paper/2310.07609/) (2023)
- [AmbigDocs: Reasoning across Documents on Different Entities under the Same Name](https://mlsanity.com/paper/2404.12447/) (2024)
- [Tanbih: Get To Know What You Are Reading](https://mlsanity.com/paper/1910.02028/) (2019)
- [Reinforcement Retrieval Leveraging Fine-grained Feedback for Fact Checking News Claims with Black-Box LLM](https://mlsanity.com/paper/2404.17283/) (2024)
- [People on Media: Jointly Identifying Credible News and Trustworthy Citizen Journalists in Online Communities](https://mlsanity.com/paper/1705.02667/) (2017)
- [Generating Media Background Checks for Automated Source Critical Reasoning](https://mlsanity.com/paper/2409.00781/) (2024)
- [MegaFake: A Theory-Driven Dataset of Fake News Generated by Large Language Models](https://mlsanity.com/paper/2408.11871/) (2024)

## Built on by (most cited first, 16 of 16)

- [Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD](https://mlsanity.com/paper/2508.17450/) (2025)
- [Retrieval- and Argumentation-Enhanced Multi-Agent LLMs for Judgmental Forecasting (Extended Version with Supplementary Material)](https://mlsanity.com/paper/2510.24303/) (2025)
- [Debating Truth: Debate-driven Claim Verification with Multiple Large Language Model Agents](https://mlsanity.com/paper/2507.19090/) (2025)
- [Courtroom-Style Multi-Agent Debate with Progressive RAG and Role-Switching for Controversial Claim Verification](https://mlsanity.com/paper/2603.28488/) (2026)
- [ConflictRAG: Detecting and Resolving Knowledge Conflicts in Retrieval Augmented Generation](https://mlsanity.com/paper/2605.17301/) (2026)
- [HaluNet: Learning Hallucination Risk from Internal Signals in LLM Question Answering](https://mlsanity.com/paper/2512.24562/) (2025)
- [The Automatic Verification of Image-Text Claims (AVerImaTeC) Shared Task](https://mlsanity.com/paper/2602.11221/) (2026)
- [Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Prediction](https://mlsanity.com/paper/2605.05244/) (2026)
- [Towards Real-Time Fake News Detection under Evidence Scarcity](https://mlsanity.com/paper/2510.11277/) (2025)
- [The Alignment Bottleneck in Decomposition-Based Claim Verification](https://mlsanity.com/paper/2602.10380/) (2026)
- [ArbGraph: Conflict-Aware Evidence Arbitration for Reliable Long-Form Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.18362/) (2026)
- [When Specifications Conflict: A Symmetry-Based Framework for Measuring LLM Preferences](https://mlsanity.com/paper/2607.28384/) (2026)
- [When Context Misleads: Intent-Guided Decoding for Robust Retrieval-Augmented Generation](https://mlsanity.com/paper/2608.16515/) (2026)
- [CoVer: Conflict-Aware Claim Verification](https://mlsanity.com/paper/2609.00508/) (2026)
- [Record Grouping Controls Evidence Weight in Language Models](https://mlsanity.com/paper/2609.08698/) (2026)
- [Multi-Channel Mitigation of Source-Trust Shortcuts in Fact-Checking RL Agents](https://mlsanity.com/paper/2609.36611/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) augmented with retrieval mechanisms have demonstrated significant potential in fact-checking tasks by integrating external knowledge. However, their reliability decreases when confronted with conflicting evidence from sources of varying credibility. This paper presents the first systematic evaluation of Retrieval-Augmented Generation (RAG) models for fact-checking in the presence of conflicting evidence. To support this study, we introduce CONFACT (Conflicting Evidence for Fact-Checking) (Dataset available at https://github.com/zoeyyes/CONFACT), a novel dataset comprising questions paired with conflicting information from various sources. Extensive experiments reveal critical vulnerabilities in state-of-the-art RAG methods, particularly in resolving conflicts stemming from differences in media source credibility. To address these challenges, we investigate strategies to integrate media background information into both the retrieval and generation stages. Our results show that effectively incorporating source credibility significantly enhances the ability of RAG models to resolve conflicting evidence and improve fact-checking performance.