# Beyond Seeing Is Believing: On Crowdsourced Detection of Audiovisual Deepfakes

arXiv: https://arxiv.org/abs/2605.04797 · HTML: https://mlsanity.com/paper/2605.04797/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 79 references that are in the map.
- First posted on arXiv 6 May 2026.

## Builds on (most cited first, 25 of 25)

- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection](https://mlsanity.com/paper/2001.03024/) (2020)
- [DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection](https://mlsanity.com/paper/2001.00179/) (2020)
- [Media Forensics and DeepFakes: an overview](https://mlsanity.com/paper/2001.06564/) (2020)
- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis](https://mlsanity.com/paper/2103.05630/) (2021)
- [KoDF: A Large-scale Korean DeepFake Detection Dataset](https://mlsanity.com/paper/2103.10094/) (2021)
- [AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset](https://mlsanity.com/paper/2311.15308/) (2023)
- [Deepfake Detection by Human Crowds, Machines, and Machine-informed Crowds](https://mlsanity.com/paper/2105.06496/) (2021)
- [Deepfake Detection: A Comprehensive Survey from the Reliability Perspective](https://mlsanity.com/paper/2211.10881/) (2022)
- [Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024](https://mlsanity.com/paper/2503.02857/) (2025)
- [On Improving Cross-dataset Generalization of Deepfake Detectors](https://mlsanity.com/paper/2204.04285/) (2022)
- [As Good As A Coin Toss: Human detection of AI-generated images, videos, audio, and audiovisual stimuli](https://mlsanity.com/paper/2403.16760/) (2024)
- [Human Detection of Political Speech Deepfakes across Transcripts, Audio, and Video](https://mlsanity.com/paper/2202.12883/) (2022)
- [PolyGlotFake: A Novel Multilingual and Multimodal DeepFake Dataset](https://mlsanity.com/paper/2405.08838/) (2024)
- [Deepfake detection: humans vs. machines](https://mlsanity.com/paper/2009.03155/) (2020)
- [Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background](https://mlsanity.com/paper/2005.06915/) (2020)
- [MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmark](https://mlsanity.com/paper/2505.11109/) (2025)
- [Trusted Media Challenge Dataset and User Study](https://mlsanity.com/paper/2201.04788/) (2022)
- [A Survey on the Role of Crowds in Combating Online Misinformation: Annotators, Evaluators, and Creators](https://mlsanity.com/paper/2310.02095/) (2023)
- [The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?](https://mlsanity.com/paper/2008.05701/) (2020)
- [The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale](https://mlsanity.com/paper/2108.01222/) (2021)
- [Can the Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation about COVID-19](https://mlsanity.com/paper/2107.11755/) (2021)
- [Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking](https://mlsanity.com/paper/2501.18265/) (2025)
- [Estimation of the qualification and behavior of a contributor and aggregation of his answers in a crowdsourcing context](https://mlsanity.com/paper/2303.04548/) (2023)

## Abstract (arXiv)

Deepfakes are increasingly realistic and easy to produce, raising concerns about the reliability of human judgments in misinformation settings. We study audiovisual deepfake detection by measuring how consistently crowd workers distinguish authentic from manipulated videos and, when they flag a video as manipulated, how accurately they identify the manipulation type (audio-only, video-only, or audio-video) and how consistently they report manipulation timestamps. We run two matched crowdsourcing studies on Prolific using AV-Deepfake1M and the Trusted Media Challenge (TMC) dataset. We sample 48 videos per dataset (96 total) and collect 960 judgments (10 per video). Results show that crowd workers rarely misclassify authentic videos as manipulated, but they miss many manipulations, and agreement remains limited across videos. Aggregating multiple judgments per video stabilizes the authenticity signal, but it cannot recover manipulations that most workers consistently miss. Manipulation type identification is substantially noisier than authenticity detection even when workers detect a manipulation, with joint audio-video cases being particularly hard to recognize. Overall, these findings suggest that crowdsourcing can provide a scalable screening signal for audiovisual authenticity, while reliable modality attribution remains an open challenge.