# CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild

arXiv: https://arxiv.org/abs/2605.30241 · HTML: https://mlsanity.com/paper/2605.30241/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 29 of 29)

- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media](https://mlsanity.com/paper/1809.01286/) (2018)
- [Factuality Challenges in the Era of Large Language Models](https://mlsanity.com/paper/2310.05189/) (2023)
- [X-FACT: A New Benchmark Dataset for Multilingual Fact Checking](https://mlsanity.com/paper/2106.09248/) (2021)
- [Generating Literal and Implied Subquestions to Fact-check Complex Claims](https://mlsanity.com/paper/2205.06938/) (2022)
- [AmbiFC: Fact-Checking Ambiguous Claims with Evidence](https://mlsanity.com/paper/2104.00640/) (2021)
- [Multi-Hop Fact Checking of Political Claims](https://mlsanity.com/paper/2009.06401/) (2020)
- [Check-COVID: Fact-Checking COVID-19 News Claims with Scientific Evidence](https://mlsanity.com/paper/2305.18265/) (2023)
- [FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation](https://mlsanity.com/paper/2410.22257/) (2024)
- [FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Models](https://mlsanity.com/paper/2502.17924/) (2025)
- [The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination](https://mlsanity.com/paper/2503.16402/) (2025)
- [A critical analysis of metrics used for measuring progress in artificial intelligence](https://mlsanity.com/paper/2008.02577/) (2020)
- [Real-time Factuality Assessment from Adversarial Feedback](https://mlsanity.com/paper/2410.14651/) (2024)
- [Community notes reduce engagement with and diffusion of false information online](https://mlsanity.com/paper/2502.13322/) (2025)
- [Did the Roll-Out of Community Notes Reduce Engagement With Misinformation on X/Twitter?](https://mlsanity.com/paper/2307.07960/) (2023)
- [MultiHoax: A Dataset of Multi-hop False-Premise Questions](https://mlsanity.com/paper/2506.00264/) (2025)
- [Breaking Language Barriers with MMTweets: Advancing Cross-Lingual Debunked Narrative Retrieval for Fact-Checking](https://mlsanity.com/paper/2308.05680/) (2023)
- [COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanations](https://mlsanity.com/paper/2510.24810/) (2025)
- [FACTors: A New Dataset for Studying the Fact-checking Ecosystem](https://mlsanity.com/paper/2505.09414/) (2025)
- [Trend of Narratives in the Age of Misinformation](https://mlsanity.com/paper/1504.05163/) (2015)
- [Independent fact-checking organizations exhibit a departure from political neutrality](https://mlsanity.com/paper/2407.19498/) (2024)
- [Consensus Stability of Community Notes on X](https://mlsanity.com/paper/2601.14002/) (2026)
- [From Birdwatch to Community Notes, from Twitter to X: four years of community-based content moderation](https://mlsanity.com/paper/2510.09585/) (2025)
- [GitSearch: Enhancing Community Notes Generation with Gap-Informed Targeted Search](https://mlsanity.com/paper/2602.08945/) (2026)
- [Grok in the Wild: Characterizing the Roles and Uses of Large Language Models on Social Media](https://mlsanity.com/paper/2602.11286/) (2026)
- [IRB: Automated Generation of Robust Factuality Benchmarks](https://mlsanity.com/paper/2602.08070/) (2026)
- [From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence](https://mlsanity.com/paper/2605.06006/) (2026)

## Abstract (arXiv)

Misinformation verification increasingly occurs in public, fast-moving, and multilingual online settings, where static benchmarks provide an incomplete measure of model reliability. We introduce CommunityFact, a refreshable benchmark for misinformation detection in the wild, with three major goals: coverage, granularity, and redistributability. This release contains 15,992 standalone claims across five languages and two domains. We evaluate ten LLMs under varying inference-time capabilities, including thinking and web-search. Our results show that closed-input verification remains challenging, web access yields the largest gains, and web-enabled LLMs' source-selection policies are systematically misaligned with the sources human Community Notes raters converge on -- a gap that closes through model-specific mechanisms of retrieval expansion or pruning. We further find substantial variation across language-domain slices and across the evidence ecosystems used by web-enabled systems. Beyond evaluation, CommunityFact positions Community Notes as a training signal for claim-conditioned source suggesters that could improve factual verification on novel claims.