# COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanations

arXiv: https://arxiv.org/abs/2510.24810 · HTML: https://mlsanity.com/paper/2510.24810/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization](https://mlsanity.com/paper/2310.16427/) (2023)
- [CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims](https://mlsanity.com/paper/2012.00614/) (2020)
- [Automated Fact-Checking for Assisting Human Fact-Checkers](https://mlsanity.com/paper/2103.07769/) (2021)
- [Generating Fact Checking Explanations](https://mlsanity.com/paper/2004.05773/) (2020)
- [A Systematic Survey of Automatic Prompt Optimization Techniques](https://mlsanity.com/paper/2502.16923/) (2025)
- [FIRE: Fact-checking with Iterative Retrieval and Verification](https://mlsanity.com/paper/2411.00784/) (2024)
- [Show Me the Work: Fact-Checkers' Requirements for Explainable Automated Fact-Checking](https://mlsanity.com/paper/2502.09083/) (2025)
- [Benchmarking the Generation of Fact Checking Explanations](https://mlsanity.com/paper/2308.15202/) (2023)
- [Supernotes: Driving Consensus in Crowd-Sourced Fact-Checking](https://mlsanity.com/paper/2411.06116/) (2024)
- [Birdwatch: Crowd Wisdom and Bridging Algorithms can Inform Understanding and Reduce the Spread of Misinformation](https://mlsanity.com/paper/2210.15723/) (2022)
- [Automated Justification Production for Claim Veracity in Fact Checking: A Survey on Architectures and Approaches](https://mlsanity.com/paper/2407.12853/) (2024)
- [SCULPT: Systematic Tuning of Long Prompts](https://mlsanity.com/paper/2410.20788/) (2024)
- [Can Community Notes Replace Professional Fact-Checkers?](https://mlsanity.com/paper/2502.14132/) (2025)
- [Diffusion of Community Fact-Checked Misinformation on Twitter](https://mlsanity.com/paper/2205.13673/) (2022)
- [Community-based fact-checking reduces the spread of misleading posts on social media](https://mlsanity.com/paper/2409.08781/) (2024)
- [Evaluating Evidence Attribution in Generated Fact Checking Explanations](https://mlsanity.com/paper/2406.12645/) (2024)
- [Community Moderation and the New Epistemology of Fact Checking on Social Media](https://mlsanity.com/paper/2505.20067/) (2025)
- [Label-semantics Aware Generative Approach for Domain-Agnostic Multilabel Classification](https://mlsanity.com/paper/2506.06806/) (2025)

## Built on by (most cited first, 5 of 5)

- [GitSearch: Enhancing Community Notes Generation with Gap-Informed Targeted Search](https://mlsanity.com/paper/2602.08945/) (2026)
- [CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild](https://mlsanity.com/paper/2605.30241/) (2026)
- [Better with Experience: Self-Evolving LLM Agents for Evidence-Grounded Health Community Notes](https://mlsanity.com/paper/2606.02215/) (2026)
- [Towards Multi-Agent-Simulation-Based Community Note Evaluation](https://mlsanity.com/paper/2606.18268/) (2026)
- [Evidence Triangulation for Multimodal Fact-Checking in the Wild](https://mlsanity.com/paper/2606.31367/) (2026)

## Abstract (arXiv)

Fact-checking on major platforms, such as X, Meta, and TikTok, is shifting from expert-driven verification to a community-based setup, where users contribute explanatory notes to clarify why a post might be misleading. An important challenge here is determining whether an explanation is helpful for understanding real-world claims and the reasons why, which remains largely underexplored in prior research. In practice, most community notes remain unpublished due to slow community annotation, and the reasons for helpfulness lack clear definitions. To bridge these gaps, we introduce the task of predicting both the helpfulness of explanatory notes and the reason for this. We present COMMUNITYNOTES, a large-scale multilingual dataset of 104k posts with user-provided notes and helpfulness labels. We further propose a framework that automatically generates and improves reason definitions via automatic prompt optimization, and integrate them into prediction. Our experiments show that the optimized definitions can improve both helpfulness and reason prediction. Finally, we show that the helpfulness information is beneficial for existing fact-checking systems.