# WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia

arXiv: https://arxiv.org/abs/2406.13805 · HTML: https://mlsanity.com/paper/2406.13805/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence](https://mlsanity.com/paper/2210.13701/) (2022)
- [Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2402.14409/) (2024)
- [FELM: Benchmarking Factuality Evaluation of Large Language Models](https://mlsanity.com/paper/2310.00741/) (2023)
- [Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation](https://mlsanity.com/paper/2210.13865/) (2022)
- [WikiContradiction: Detecting Self-Contradiction Articles on Wikipedia](https://mlsanity.com/paper/2111.08543/) (2021)
- [Missci: Reconstructing Fallacies in Misrepresented Science](https://mlsanity.com/paper/2406.03181/) (2024)
- [Defending Against Disinformation Attacks in Open-Domain Question Answering](https://mlsanity.com/paper/2212.10002/) (2022)

## Built on by (most cited first, 30 of 44)

- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models](https://mlsanity.com/paper/2506.01062/) (2025)
- [DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs](https://mlsanity.com/paper/2506.08500/) (2025)
- [Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models](https://mlsanity.com/paper/2505.19509/) (2025)
- [Task Matters: Knowledge Requirements Shape LLM Responses to Context-Memory Conflict](https://mlsanity.com/paper/2506.06485/) (2025)
- [MAGIC: A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts in Retrieval-Augmented Generation](https://mlsanity.com/paper/2507.21544/) (2025)
- [NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systems](https://mlsanity.com/paper/2601.11004/) (2026)
- [Consensus or Conflict? Fine-Grained Evaluation of Conflicting Answers in Question-Answering](https://mlsanity.com/paper/2508.12355/) (2025)
- [Wikipedia in the Era of LLMs: Evolution and Risks](https://mlsanity.com/paper/2503.02879/) (2025)
- [Format as a Prior: Quantifying and Analyzing Bias in LLMs for Heterogeneous Data](https://mlsanity.com/paper/2508.15793/) (2025)
- [Detecting Corpus-Level Knowledge Inconsistencies in Wikipedia with Large Language Models](https://mlsanity.com/paper/2509.23233/) (2025)
- [PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies](https://mlsanity.com/paper/2510.16505/) (2025)
- [KScope: A Framework for Characterizing the Knowledge Status of Language Models](https://mlsanity.com/paper/2506.07458/) (2025)
- [HealthContradict: Evaluating Biomedical Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2512.02299/) (2025)
- [Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation](https://mlsanity.com/paper/2510.07926/) (2025)
- [MemEIC: A Step Toward Continual and Compositional Knowledge Editing](https://mlsanity.com/paper/2510.25798/) (2025)
- [Heaven-Sent or Hell-Bent? Benchmarking the Intelligence and Defectiveness of LLM Hallucinations](https://mlsanity.com/paper/2512.21635/) (2025)
- [Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistency](https://mlsanity.com/paper/2601.05905/) (2026)
- [Mitigating Context-Memory Conflicts in LLMs through Dynamic Cognitive Reconciliation Decoding](https://mlsanity.com/paper/2605.12185/) (2026)
- [Detecting Is Not Resolving: The Monitoring Control Gap in Retrieval Augmented LLMs](https://mlsanity.com/paper/2605.27157/) (2026)
- [Magic Mushroom: A Customizable Benchmark for Fine-grained Analysis of Retrieval Noise Erosion in RAG Systems](https://mlsanity.com/paper/2506.03901/) (2025)
- [CryptoAnalystBench: Failures in Multi-Tool Long-Form LLM Analysis](https://mlsanity.com/paper/2602.11304/) (2026)
- [LLM-Oriented Information Retrieval: A Denoising-First Perspective](https://mlsanity.com/paper/2605.00505/) (2026)
- [EditPropBench: Measuring Factual Edit Propagation in Scientific Manuscripts](https://mlsanity.com/paper/2605.02083/) (2026)
- [Trust or Abstain? A Self-Aware RAG Approach](https://mlsanity.com/paper/2605.18792/) (2026)
- [Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluation](https://mlsanity.com/paper/2607.16019/) (2026)
- [Information Discernment in Large Language Models](https://mlsanity.com/paper/2607.19355/) (2026)
- [When Personal Memory Has No Single Answer: Evaluating LLM Agents under Irreducible Conflict](https://mlsanity.com/paper/2608.13921/) (2026)
- [A Course Shared Task on Evaluating LLM Output for Clinical Questions](https://mlsanity.com/paper/2408.00122/) (2024)
- [On Finding Inconsistencies in Documents](https://mlsanity.com/paper/2512.18601/) (2025)

## Abstract (arXiv)

Retrieval-augmented generation (RAG) has emerged as a promising solution to mitigate the limitations of large language models (LLMs), such as hallucinations and outdated information. However, it remains unclear how LLMs handle knowledge conflicts arising from different augmented retrieved passages, especially when these passages originate from the same source and have equal trustworthiness. In this work, we conduct a comprehensive evaluation of LLM-generated answers to questions that have varying answers based on contradictory passages from Wikipedia, a dataset widely regarded as a high-quality pre-training resource for most LLMs. Specifically, we introduce WikiContradict, a benchmark consisting of 253 high-quality, human-annotated instances designed to assess LLM performance when augmented with retrieved passages containing real-world knowledge conflicts. We benchmark a diverse range of both closed and open-source LLMs under different QA scenarios, including RAG with a single passage, and RAG with 2 contradictory passages. Through rigorous human evaluations on a subset of WikiContradict instances involving 5 LLMs and over 3,500 judgements, we shed light on the behaviour and limitations of these models. For instance, when provided with two passages containing contradictory facts, all models struggle to generate answers that accurately reflect the conflicting nature of the context, especially for implicit conflicts requiring reasoning. Since human evaluation is costly, we also introduce an automated model that estimates LLM performance using a strong open-source language model, achieving an F-score of 0.8. Using this automated metric, we evaluate more than 1,500 answers from seven LLMs across all WikiContradict instances. To facilitate future work, we release WikiContradict on: https://ibm.biz/wikicontradict.