Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking
Not on the map. Why?
Abstract (arXiv)
Evaluating the truthfulness of online content is critical for combating misinformation. This study examines the efficiency and effectiveness of crowdsourced truthfulness assessments through a comparative analysis of two approaches: one involving full-length webpages as evidence for each claim, and another using summaries for each evidence document generated with a large language model. Using an A/B testing setting, we engage a diverse pool of participants tasked with evaluating the truthfulness of statements under these conditions. Our analysis explores both the quality of assessments and the behavioral patterns of participants. The results reveal that relying on summarized evidence offers comparable accuracy and error metrics to the Standard modality while significantly improving efficiency. Workers in the Summary setting complete a significantly higher number of assessments, reducing task duration and costs. Additionally, the Summary modality maximizes internal agreement and maintains consistent reliance on and perceived usefulness of evidence, demonstrating its potential to streamline large-scale truthfulness evaluations.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 21 of its 64 references that are in the map. First posted on arXiv 30 Jan 2025.
Builds on
21 of 64 references.
References · 21
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
- 2019MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claimscited by 169
- 2020Generating Fact Checking Explanationscited by 132
- 2023Factuality Challenges in the Era of Large Language Modelscited by 125
- 2022End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Modelscited by 105
- 2023The science of fake newscited by 94
- 2023Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkerscited by 86
- 2024The Automated Verification of Textual Claims (AVeriTeC) Shared Taskcited by 29
- 2017A simple but tough-to-beat baseline for the Fake News Challenge stance detection taskcited by 26
- 2024Claim Verification in the Age of Large Language Models: A Surveycited by 23
- 2021Detecting Fake News Using Machine Learning : A Systematic Literature Reviewcited by 19
9 more
- 2022Crowdsourced Fact-Checking at Twitter: How Does the Crowd Compare With Experts?cited by 16
- 2019Learning from Fact-checkers: Analysis and Generation of Fact-checking Languagecited by 14
- 2024Surprising Efficacy of Fine-Tuned Transformers for Fact-Checking over Larger Language Modelscited by 14
- 2022Fact Checking with Insufficient Evidencecited by 11
- 2020Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Backgroundcited by 9
- 2023Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AIcited by 9
- 2021The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scalecited by 5
- 2021Scalable Fact-checking with Human-in-the-Loopcited by 5
- 2024Exploring Multidimensional Checkworthiness: Designing AI-assisted Claim Prioritization for Human Fact-checkerscited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.