# Seeing, Hearing, and Knowing Together: Multimodal Strategies in Deepfake Videos Detection

arXiv: https://arxiv.org/abs/2602.01284 · HTML: https://mlsanity.com/paper/2602.01284/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 42 references that are in the map.
- First posted on arXiv 1 Feb 2026.

## Builds on (most cited first, 10 of 10)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [The Creation and Detection of Deepfakes: A Survey](https://mlsanity.com/paper/2004.11138/) (2020)
- [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://mlsanity.com/paper/1811.07901/) (2018)
- [AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection](https://mlsanity.com/paper/2406.02951/) (2024)
- [Deepfake Detection by Human Crowds, Machines, and Machine-informed Crowds](https://mlsanity.com/paper/2105.06496/) (2021)
- [Human Detection of Political Speech Deepfakes across Transcripts, Audio, and Video](https://mlsanity.com/paper/2202.12883/) (2022)
- [Testing Human Ability To Detect Deepfake Images of Human Faces](https://mlsanity.com/paper/2212.05056/) (2022)
- [The Value of AI Guidance in Human Examination of Synthetically-Generated Faces](https://mlsanity.com/paper/2208.10544/) (2022)
- [Diverse Misinformation: Impacts of Human Biases on Detection of Deepfakes on Networks](https://mlsanity.com/paper/2210.10026/) (2022)

## Built on by (most cited first, 1 of 1)

- [Detecting AI-Generated Video: A Vision-Language Dual-View Survey](https://mlsanity.com/paper/2607.10787/) (2026)

## Abstract (arXiv)

As deepfake videos become increasingly difficult for people to recognise, understanding the strategies humans use is key to designing effective media literacy interventions. We conducted a study with 195 participants between the ages of 21 and 40, who judged real and deepfake videos, rated their confidence, and reported the cues they relied on across visual, audio, and knowledge strategies. Participants were more accurate with real videos than with deepfakes and showed lower expected calibration error for real content. Through association rule mining, we identified cue combinations that shaped performance. Visual appearance, vocal, and intuition often co-occurred for successful identifications, which highlights the importance of multimodal approaches in human detection. Our findings show which cues help or hinder detection and suggest directions for designing media literacy tools that guide effective cue use. Building on these insights can help people improve their identification skills and become more resilient to deceptive digital media.