# Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking

arXiv: https://arxiv.org/abs/2501.18265 · HTML: https://mlsanity.com/paper/2501.18265/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 64 references that are in the map.
- First posted on arXiv 30 Jan 2025.

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims](https://mlsanity.com/paper/1909.03242/) (2019)
- [Generating Fact Checking Explanations](https://mlsanity.com/paper/2004.05773/) (2020)
- [Factuality Challenges in the Era of Large Language Models](https://mlsanity.com/paper/2310.05189/) (2023)
- [End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models](https://mlsanity.com/paper/2205.12487/) (2022)
- [The science of fake news](https://mlsanity.com/paper/2307.07903/) (2023)
- [Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers](https://mlsanity.com/paper/2311.09000/) (2023)
- [The Automated Verification of Textual Claims (AVeriTeC) Shared Task](https://mlsanity.com/paper/2410.23850/) (2024)
- [A simple but tough-to-beat baseline for the Fake News Challenge stance detection task](https://mlsanity.com/paper/1707.03264/) (2017)
- [Claim Verification in the Age of Large Language Models: A Survey](https://mlsanity.com/paper/2408.14317/) (2024)
- [Detecting Fake News Using Machine Learning : A Systematic Literature Review](https://mlsanity.com/paper/2102.04458/) (2021)
- [Crowdsourced Fact-Checking at Twitter: How Does the Crowd Compare With Experts?](https://mlsanity.com/paper/2208.09214/) (2022)
- [Learning from Fact-checkers: Analysis and Generation of Fact-checking Language](https://mlsanity.com/paper/1910.02202/) (2019)
- [Surprising Efficacy of Fine-Tuned Transformers for Fact-Checking over Larger Language Models](https://mlsanity.com/paper/2402.12147/) (2024)
- [Fact Checking with Insufficient Evidence](https://mlsanity.com/paper/2204.02007/) (2022)
- [Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background](https://mlsanity.com/paper/2005.06915/) (2020)
- [Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI](https://mlsanity.com/paper/2308.07213/) (2023)
- [The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale](https://mlsanity.com/paper/2108.01222/) (2021)
- [Scalable Fact-checking with Human-in-the-Loop](https://mlsanity.com/paper/2109.10992/) (2021)
- [Exploring Multidimensional Checkworthiness: Designing AI-assisted Claim Prioritization for Human Fact-checkers](https://mlsanity.com/paper/2412.08185/) (2024)

## Built on by (most cited first, 2 of 2)

- [The Effect of Document Summarization on LLM-Based Relevance Judgments](https://mlsanity.com/paper/2512.05334/) (2025)
- [Beyond Seeing Is Believing: On Crowdsourced Detection of Audiovisual Deepfakes](https://mlsanity.com/paper/2605.04797/) (2026)

## Abstract (arXiv)

Evaluating the truthfulness of online content is critical for combating misinformation. This study examines the efficiency and effectiveness of crowdsourced truthfulness assessments through a comparative analysis of two approaches: one involving full-length webpages as evidence for each claim, and another using summaries for each evidence document generated with a large language model. Using an A/B testing setting, we engage a diverse pool of participants tasked with evaluating the truthfulness of statements under these conditions. Our analysis explores both the quality of assessments and the behavioral patterns of participants. The results reveal that relying on summarized evidence offers comparable accuracy and error metrics to the Standard modality while significantly improving efficiency. Workers in the Summary setting complete a significantly higher number of assessments, reducing task duration and costs. Additionally, the Summary modality maximizes internal agreement and maintains consistent reliance on and perceived usefulness of evidence, demonstrating its potential to streamline large-scale truthfulness evaluations.