# Integrating Audio-Visual Features for Multimodal Deepfake Detection

arXiv: https://arxiv.org/abs/2310.03827 · HTML: https://mlsanity.com/paper/2310.03827/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019)
- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- [Multi-attentional Deepfake Detection](https://mlsanity.com/paper/2103.02406/) (2021)
- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [Exploring Temporal Coherence for More General Video Face Forgery Detection](https://mlsanity.com/paper/2108.06693/) (2021)
- [FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces](https://mlsanity.com/paper/1803.09179/) (2018)
- [Self-Supervised Video Forensics by Audio-Visual Anomaly Detection](https://mlsanity.com/paper/2301.01767/) (2023)
- [KoDF: A Large-scale Korean DeepFake Detection Dataset](https://mlsanity.com/paper/2103.10094/) (2021)
- [Use of a Capsule Network to Detect Fake Images and Videos](https://mlsanity.com/paper/1910.12467/) (2019)
- [Voice-Face Homogeneity Tells Deepfake](https://mlsanity.com/paper/2203.02195/) (2022)
- [Audio-Visual Person-of-Interest DeepFake Detection](https://mlsanity.com/paper/2204.03083/) (2022)
- [AI-Synthesized Voice Detection Using Neural Vocoder Artifacts](https://mlsanity.com/paper/2304.13085/) (2023)
- [Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors](https://mlsanity.com/paper/2109.02993/) (2021)
- [Level Up the Deepfake Detection: a Method to Effectively Discriminate Images Generated by GAN Architectures and Diffusion Models](https://mlsanity.com/paper/2303.00608/) (2023)
- [Model Attribution of Face-swap Deepfake Videos](https://mlsanity.com/paper/2202.12951/) (2022)
- [Trusted Media Challenge Dataset and User Study](https://mlsanity.com/paper/2201.04788/) (2022)

## Built on by (most cited first, 9 of 9)

- [Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes](https://mlsanity.com/paper/2401.15668/) (2024)
- [From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape](https://mlsanity.com/paper/2312.10868/) (2023)
- [Exposing Lip-syncing Deepfakes from Mouth Inconsistencies](https://mlsanity.com/paper/2401.10113/) (2024)
- [AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection](https://mlsanity.com/paper/2310.13103/) (2023)
- [Detecting Lip-Syncing Deepfakes: Vision Temporal Transformer for Analyzing Mouth Inconsistencies](https://mlsanity.com/paper/2504.01470/) (2025)
- [Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges](https://mlsanity.com/paper/2406.06965/) (2024)
- [DeepFake-O-Meter v2.0: An Open Platform for DeepFake Detection](https://mlsanity.com/paper/2404.13146/) (2024)
- [Explicit Correlation Learning for Generalizable Cross-Modal Deepfake Detection](https://mlsanity.com/paper/2404.19171/) (2024)
- [A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection](https://mlsanity.com/paper/2406.14176/) (2024)

## Abstract (arXiv)

Deepfakes are AI-generated media in which an image or video has been digitally modified. The advancements made in deepfake technology have led to privacy and security issues. Most deepfake detection techniques rely on the detection of a single modality. Existing methods for audio-visual detection do not always surpass that of the analysis based on single modalities. Therefore, this paper proposes an audio-visual-based method for deepfake detection, which integrates fine-grained deepfake identification with binary classification. We categorize the samples into four types by combining labels specific to each single modality. This method enhances the detection under intra-domain and cross-domain testing.