mlsanityData to 14 Oct 2026

Detecting Corpus-Level Knowledge Inconsistencies in Wikipedia with Large Language Models

arXiv 2509.23233 ↗First posted 27 Sep 2025cs.CLcited by 4

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Wikipedia is the largest open knowledge corpus, widely used worldwide and serving as a key resource for training large language models (LLMs) and retrieval-augmented generation (RAG) systems. Ensuring its accuracy is therefore critical. But how accurate is Wikipedia, and how can we improve it? We focus on inconsistencies, a specific type of factual inaccuracy, and introduce the task of corpus-level inconsistency detection. We present CLAIRE, an agentic system that combines LLM reasoning with retrieval to surface potentially inconsistent claims along with contextual evidence for human review. In a user study with experienced Wikipedia editors, 87.5% reported higher confidence when using CLAIRE, and participants identified 64.7% more inconsistencies in the same amount of time. Combining CLAIRE with human annotation, we contribute WIKICOLLIDE, the first benchmark of real Wikipedia inconsistencies. Using random sampling with CLAIRE-assisted analysis, we find that at least 3.3% of English Wikipedia facts contradict another fact, with inconsistencies propagating into 7.3% of FEVEROUS and 4.0% of AmbigQA examples. Benchmarking strong baselines on this dataset reveals substantial headroom: the best fully automated system achieves an AUROC of only 75.1%. Our results show that contradictions are a measurable component of Wikipedia and that LLM-based systems like CLAIRE can provide a practical tool to help editors improve knowledge consistency at scale.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 36 of its 81 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

36 of 81 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Retrieval-Augmented Generation with Conflicting Evidence17 Apr
  2. 2024WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia19 Jun
  3. 2024Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Models22 Feb

Other references · 33

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  6. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  8. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  9. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  10. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  11. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  12. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
21 more
  1. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  2. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  3. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 201
  4. 2021FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured informationcited by 167
  5. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  6. 2018The Fact Extraction and VERification (FEVER) Shared Taskcited by 108
  7. 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
  8. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  9. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 44
  10. 2023EX-FEVER: A Dataset for Multi-hop Explainable Fact Verificationcited by 36
  11. 2024Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance?cited by 36
  12. 2024DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Modelscited by 34
  13. 2024The Automated Verification of Textual Claims (AVeriTeC) Shared Taskcited by 29
  14. 2021WikiContradiction: Detecting Self-Contradiction Articles on Wikipediacited by 17
  15. 2024Improving Model Factuality with Fine-grained Critique-based Evaluatorcited by 15
  16. 2024Localizing Factual Inconsistencies in Attributable Text Generationcited by 14
  17. 2024SPAGHETTI: Open-Domain Question Answering from Heterogeneous Data Sources with Retrieval and Semantic Parsingcited by 11
  18. 2025Verify-in-the-Graph: Entity Disambiguation Enhancement for Complex Claim Verification with Interactive Graph Representationcited by 5
  19. 2024AMREx: AMR for Explainable Fact Verificationcited by 4
  20. 2024ZeFaV: Boosting Large Language Models for Zero-shot Fact Verificationcited by 1
  21. 2024Self-Adaptive Paraphrasing and Preference Learning for Improved Claim Verifiabilitycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environmentscited by 3
  2. 2025On Finding Inconsistencies in Documentscited by 0
  3. 2026SpectraQuery: A Hybrid Retrieval-Augmented Conversational Assistant for Battery Sciencecited by 0
  4. 2026Detecting Knowledge Inconsistencies Across Text, Tables, and Knowledge Graphscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond
  2. 2025Understanding Inequality of LLM Fact-Checking over Geographic Regions with Agent and Retrieval models
  3. 2023Fine-tuning Language Models for Factuality
  4. 2026Fair Fact-Checking: Closing the Cross-Lingual Gap in LLM Factual Judgement with RoSh
  5. 2026Can Language Models Play the Wikipedia Game?
  6. 2024Language agents achieve superhuman synthesis of scientific knowledge
  7. 2023FELM: Benchmarking Factuality Evaluation of Large Language Models
  8. 2024UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Models