# Unmasking Illusions: Understanding Human Perception of Audiovisual Deepfakes

arXiv: https://arxiv.org/abs/2405.04097 · HTML: https://mlsanity.com/paper/2405.04097/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 73 references that are in the map.
- First posted on arXiv 7 May 2024.

## Builds on (most cited first, 29 of 29)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- [FSGAN: Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/1908.05932/) (2019)
- [Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection](https://mlsanity.com/paper/2012.07657/) (2020)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [Fast Face-swap Using Convolutional Neural Networks](https://mlsanity.com/paper/1611.09577/) (2016)
- [Deep Learning for Deepfakes Creation and Detection: A Survey](https://mlsanity.com/paper/1909.11573/) (2019)
- [Seeing is not always believing: Benchmarking Human and Model Perception of AI-Generated Images](https://mlsanity.com/paper/2304.13023/) (2023)
- [HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models](https://mlsanity.com/paper/1904.01121/) (2019)
- [Deepfake Detection by Human Crowds, Machines, and Machine-informed Crowds](https://mlsanity.com/paper/2105.06496/) (2021)
- [An Introduction to Image Synthesis with Generative Adversarial Nets](https://mlsanity.com/paper/1803.04469/) (2018)
- [Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding](https://mlsanity.com/paper/2005.08484/) (2020)
- [A Representative Study on Human Detection of Artificially Generated Media Across Countries](https://mlsanity.com/paper/2312.05976/) (2023)
- [As Good As A Coin Toss: Human detection of AI-generated images, videos, audio, and audiovisual stimuli](https://mlsanity.com/paper/2403.16760/) (2024)
- [Warning: Humans Cannot Reliably Detect Speech Deepfakes](https://mlsanity.com/paper/2301.07829/) (2023)
- [Human Perception of Audio Deepfakes](https://mlsanity.com/paper/2107.09667/) (2021)
- [Testing Human Ability To Detect Deepfake Images of Human Faces](https://mlsanity.com/paper/2212.05056/) (2022)
- [A Study of the Human Perception of Synthetic Faces](https://mlsanity.com/paper/2111.04230/) (2021)
- [Humans Are Easily Fooled by Digital Images](https://mlsanity.com/paper/1509.05301/) (2015)
- [AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection](https://mlsanity.com/paper/2310.13103/) (2023)
- [Can you hear me textitnow? Sensitive comparisons of human and machine perception](https://mlsanity.com/paper/2003.12362/) (2020)
- [Audio Deepfake Perceptions in College Going Populations](https://mlsanity.com/paper/2112.03351/) (2021)

## Built on by (most cited first, 7 of 7)

- [Deepfake Detection that Generalizes Across Benchmarks](https://mlsanity.com/paper/2508.06248/) (2025)
- [How Good is ChatGPT at Audiovisual Deepfake Detection: A Comparative Study of ChatGPT, AI Models and Human Perception](https://mlsanity.com/paper/2411.09266/) (2024)
- [Signals of Provenance: Practices & Challenges of Navigating Indicators in AI-Generated Media for Sighted and Blind Individuals](https://mlsanity.com/paper/2505.16057/) (2025)
- [Personhood Credentials: Human-Centered Design Recommendation Balancing Security, Usability, and Trust](https://mlsanity.com/paper/2502.16375/) (2025)
- [FakeRadar: Probing Forgery Outliers to Detect Unknown Deepfake Videos](https://mlsanity.com/paper/2512.14601/) (2025)
- [Human-AI Ensembles Improve Deepfake Detection in Low-to-Medium Quality Videos](https://mlsanity.com/paper/2603.14658/) (2026)
- [I Hear, Therefore I Trust: A Socio-Technical Investigation of Humans as Synthetic Speech Detectors](https://mlsanity.com/paper/2605.28064/) (2026)

## Abstract (arXiv)

The emergence of contemporary deepfakes has attracted significant attention in machine learning research, as artificial intelligence (AI) generated synthetic media increases the incidence of misinterpretation and is difficult to distinguish from genuine content. Currently, machine learning techniques have been extensively studied for automatically detecting deepfakes. However, human perception has been less explored. Malicious deepfakes could ultimately cause public and social problems. Can we humans correctly perceive the authenticity of the content of the videos we watch? The answer is obviously uncertain; therefore, this paper aims to evaluate the human ability to discern deepfake videos through a subjective study. We present our findings by comparing human observers to five state-ofthe-art audiovisual deepfake detection models. To this end, we used gamification concepts to provide 110 participants (55 native English speakers and 55 non-native English speakers) with a webbased platform where they could access a series of 40 videos (20 real and 20 fake) to determine their authenticity. Each participant performed the experiment twice with the same 40 videos in different random orders. The videos are manually selected from the FakeAVCeleb dataset. We found that all AI models performed better than humans when evaluated on the same 40 videos. The study also reveals that while deception is not impossible, humans tend to overestimate their detection capabilities. Our experimental results may help benchmark human versus machine performance, advance forensics analysis, and enable adaptive countermeasures.