# Large Language Models Require Curated Context for Reliable Political Fact-Checking -- Even with Reasoning and Web Search

arXiv: https://arxiv.org/abs/2511.18749 · HTML: https://mlsanity.com/paper/2511.18749/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 25 of its 66 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 25 of 25)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims](https://mlsanity.com/paper/1909.03242/) (2019)
- [Fact-Checking Complex Claims with Program-Guided Reasoning](https://mlsanity.com/paper/2305.12744/) (2023)
- [Factuality Challenges in the Era of Large Language Models](https://mlsanity.com/paper/2310.05189/) (2023)
- [Combining Fact Extraction and Verification with Neural Semantic Matching Networks](https://mlsanity.com/paper/1811.07039/) (2018)
- [Complex Claim Verification with Evidence Retrieved in the Wild](https://mlsanity.com/paper/2305.11859/) (2023)
- [Measuring Political Bias in Large Language Models: What Is Said and How It Is Said](https://mlsanity.com/paper/2403.18932/) (2024)
- [Explainable Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2011.03870/) (2020)
- [RumourEval 2019: Determining Rumour Veracity and Support for Rumours](https://mlsanity.com/paper/1809.06683/) (2018)
- [CREPE: Open-Domain Question Answering with False Presuppositions](https://mlsanity.com/paper/2211.17257/) (2022)
- [BERT for Evidence Retrieval and Claim Verification](https://mlsanity.com/paper/1910.02655/) (2019)
- [Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2109.11427/) (2021)
- [Language Models as Fact Checkers?](https://mlsanity.com/paper/2006.04102/) (2020)
- [When Large Language Models Meet Vector Databases: A Survey](https://mlsanity.com/paper/2402.01763/) (2024)
- [On the Relationship between Truth and Political Bias in Language Models](https://mlsanity.com/paper/2409.05283/) (2024)
- [Fact-checking information from large language models can decrease headline discernment](https://mlsanity.com/paper/2308.10800/) (2023)
- [Supernotes: Driving Consensus in Crowd-Sourced Fact-Checking](https://mlsanity.com/paper/2411.06116/) (2024)
- [News Source Citing Patterns in AI Search Systems](https://mlsanity.com/paper/2507.05301/) (2025)
- [NELA-Local: A Dataset of U.S. Local News Articles for the Study of County-level News Ecosystems](https://mlsanity.com/paper/2203.08600/) (2022)
- [Evaluating open-source Large Language Models for automated fact-checking](https://mlsanity.com/paper/2503.05565/) (2025)
- [DomainDemo: a dataset of domain-sharing activities among different demographic groups on Twitter](https://mlsanity.com/paper/2501.09035/) (2025)

## Built on by (most cited first, 2 of 2)

- [GitSearch: Enhancing Community Notes Generation with Gap-Informed Targeted Search](https://mlsanity.com/paper/2602.08945/) (2026)
- [Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answers](https://mlsanity.com/paper/2602.22221/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have raised hopes for automated end-to-end fact-checking, but prior studies report mixed results. As mainstream chatbots increasingly ship with reasoning capabilities and web search tools -- and millions of users already rely on them for verification -- rigorous evaluation is urgent. We evaluate 15 recent LLMs from OpenAI, Google, Meta, and DeepSeek on more than 6,000 claims fact-checked by PolitiFact, comparing standard models with reasoning- and web-search variants. Standard models perform poorly, reasoning offers minimal benefits, and web search provides only moderate gains, despite fact-checks being available on the web. In contrast, a curated RAG system using PolitiFact summaries improved macro F1 by 233% on average across model variants. These findings suggest that giving models access to curated high-quality context is a promising path for automated fact-checking.