# Crowdsourced Fact-Checking at Twitter: How Does the Crowd Compare With Experts?

arXiv: https://arxiv.org/abs/2208.09214 · HTML: https://mlsanity.com/paper/2208.09214/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 58 references that are in the map.
- First posted on arXiv 19 Aug 2022.

## Builds on (most cited first, 18 of 18)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Automated Fact-Checking for Assisting Human Fact-Checkers](https://mlsanity.com/paper/2103.07769/) (2021)
- [The Fact Extraction and VERification (FEVER) Shared Task](https://mlsanity.com/paper/1811.10971/) (2018)
- [Automated Fact Checking: Task formulations, methods and future directions](https://mlsanity.com/paper/1806.07687/) (2018)
- [That is a Known Lie: Detecting Previously Fact-Checked Claims](https://mlsanity.com/paper/2005.06058/) (2020)
- [Generating Fact Checking Briefs](https://mlsanity.com/paper/2011.05448/) (2020)
- [The Rise of Guardians: Fact-checking URL Recommendation to Combat Fake News](https://mlsanity.com/paper/1806.07516/) (2018)
- [The Role of the Crowd in Countering Misinformation: A Case Study of the COVID-19 Infodemic](https://mlsanity.com/paper/2011.05773/) (2020)
- [BRENDA: Browser Extension for Fake News Detection](https://mlsanity.com/paper/2005.13270/) (2020)
- [Modeling Conversation Structure and Temporal Dynamics for Jointly Predicting Rumor Stance and Veracity](https://mlsanity.com/paper/1909.08211/) (2019)
- [Community-Based Fact-Checking on Twitter's Birdwatch Platform](https://mlsanity.com/paper/2104.07175/) (2021)
- [Neural Check-Worthiness Ranking with Weak Supervision: Finding Sentences for Fact-Checking](https://mlsanity.com/paper/1903.08404/) (2019)
- [Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background](https://mlsanity.com/paper/2005.06915/) (2020)
- [Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria](https://mlsanity.com/paper/2008.09533/) (2020)
- [Unsupervised Matching of Data and Text](https://mlsanity.com/paper/2112.08776/) (2021)
- [Scrutinizer: A Mixed-Initiative Approach to Large-Scale, Data-Driven Claim Verification](https://mlsanity.com/paper/2003.06708/) (2020)
- [The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?](https://mlsanity.com/paper/2008.05701/) (2020)
- [Real-time Claim Detection from News Articles and Retrieval of Semantically-Similar Factchecks](https://mlsanity.com/paper/1907.02030/) (2019)

## Built on by (most cited first, 16 of 16)

- [MOSAIC: Modeling Social AI for Content Dissemination and Regulation in Multi-Agent Simulations](https://mlsanity.com/paper/2504.07830/) (2025)
- [Zero-shot Faithful Factual Error Correction](https://mlsanity.com/paper/2305.07982/) (2023)
- [The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Who](https://mlsanity.com/paper/2304.14238/) (2023)
- [PCoT: Persuasion-Augmented Chain of Thought for Detecting Fake News and Social Media Disinformation](https://mlsanity.com/paper/2506.06842/) (2025)
- [Can Community Notes Replace Professional Fact-Checkers?](https://mlsanity.com/paper/2502.14132/) (2025)
- [VeraCT Scan: Retrieval-Augmented Fake News Detection with Justifiable Reasoning](https://mlsanity.com/paper/2406.10289/) (2024)
- [Community Moderation and the New Epistemology of Fact Checking on Social Media](https://mlsanity.com/paper/2505.20067/) (2025)
- [False Information, Bots and Malicious Campaigns: Demystifying Elements of Social Media Manipulations](https://mlsanity.com/paper/2308.12497/) (2023)
- [Fine-Grained Appropriate Reliance: Human-AI Collaboration with a Multi-Step Transparent Decision Workflow for Complex Task Decomposition](https://mlsanity.com/paper/2501.10909/) (2025)
- [Assessing the Potential of Generative Agents in Crowdsourced Fact-Checking](https://mlsanity.com/paper/2504.19940/) (2025)
- [Stand for Something or Fall for Everything: Predict Misinformation Spread with Stance-Aware Graph Neural Networks](https://mlsanity.com/paper/2310.02568/) (2023)
- [Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking](https://mlsanity.com/paper/2501.18265/) (2025)
- [Combating Misinformation in the Arab World: Challenges & Opportunities](https://mlsanity.com/paper/2506.05582/) (2025)
- [Automated Alignment between Elicitation Interviews and Requirements](https://mlsanity.com/paper/2510.08622/) (2025)
- [Analyzing Sustainability Messaging in Large-Scale Corporate Social Media](https://mlsanity.com/paper/2511.01550/) (2025)
- [ExTax: Explainable Disinformation Detection via Persuasion, Emotion, and Narrative Role Taxonomies](https://mlsanity.com/paper/2605.27045/) (2026)

## Abstract (arXiv)

Fact-checking is one of the effective solutions in fighting online misinformation. However, traditional fact-checking is a process requiring scarce expert human resources, and thus does not scale well on social media because of the continuous flow of new content to be checked. Methods based on crowdsourcing have been proposed to tackle this challenge, as they can scale with a smaller cost, but, while they have shown to be feasible, have always been studied in controlled environments. In this work, we study the first large-scale effort of crowdsourced fact-checking deployed in practice, started by Twitter with the Birdwatch program. Our analysis shows that crowdsourcing may be an effective fact-checking strategy in some settings, even comparable to results obtained by human experts, but does not lead to consistent, actionable results in others. We processed 11.9k tweets verified by the Birdwatch program and report empirical evidence of i) differences in how the crowd and experts select content to be fact-checked, ii) how the crowd and the experts retrieve different resources to fact-check, and iii) the edge the crowd shows in fact-checking scalability and efficiency as compared to expert checkers.