# Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors

arXiv: https://arxiv.org/abs/2109.02993 · HTML: https://mlsanity.com/paper/2109.02993/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 99 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Builds on (most cited first, 30 of 51)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- [FSGAN: Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/1908.05932/) (2019)
- [Exposing Deep Fakes Using Inconsistent Head Poses](https://mlsanity.com/paper/1811.00661/) (2018)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection](https://mlsanity.com/paper/2001.03024/) (2020)
- [Neural Voice Puppetry: Audio-driven Facial Reenactment](https://mlsanity.com/paper/1912.05566/) (2019)
- [DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection](https://mlsanity.com/paper/2001.00179/) (2020)
- [FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping](https://mlsanity.com/paper/1912.13457/) (2019)
- [DeepFakes: a New Threat to Face Recognition? Assessment and Detection](https://mlsanity.com/paper/1812.08685/) (2018)
- [WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection](https://mlsanity.com/paper/2101.01456/) (2021)
- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection](https://mlsanity.com/paper/1904.05441/) (2019)
- [Searching Central Difference Convolutional Networks for Face Anti-Spoofing](https://mlsanity.com/paper/2003.04092/) (2020)
- [Learning Rich Features for Image Manipulation Detection](https://mlsanity.com/paper/1805.04953/) (2018)
- [Two-Stream Neural Networks for Tampered Face Detection](https://mlsanity.com/paper/1803.11276/) (2018)
- [Fast Face-swap Using Convolutional Neural Networks](https://mlsanity.com/paper/1611.09577/) (2016)
- [Recurrent Convolutional Strategies for Face Manipulation Detection in Videos](https://mlsanity.com/paper/1905.00582/) (2019)
- [Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using Affective Cues](https://mlsanity.com/paper/2003.06711/) (2020)
- [ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection](https://mlsanity.com/paper/1812.02510/) (2018)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization](https://mlsanity.com/paper/2005.14405/) (2020)

## Built on by (most cited first, 30 of 32)

- [Detecting Multimedia Generated by Large AI Models: A Survey](https://mlsanity.com/paper/2402.00045/) (2024)
- [Voice-Face Homogeneity Tells Deepfake](https://mlsanity.com/paper/2203.02195/) (2022)
- [Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization](https://mlsanity.com/paper/2305.01979/) (2023)
- [Combating Online Misinformation Videos: Characterization, Detection, and Future Directions](https://mlsanity.com/paper/2302.03242/) (2023)
- [SoK: Systematization and Benchmarking of Deepfake Detectors in a Unified Framework](https://mlsanity.com/paper/2401.04364/) (2024)
- [MIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detection](https://mlsanity.com/paper/2310.02234/) (2023)
- [Integrating Audio-Visual Features for Multimodal Deepfake Detection](https://mlsanity.com/paper/2310.03827/) (2023)
- [FakeOut: Leveraging Out-of-domain Self-supervision for Multi-modal Video Deepfake Detection](https://mlsanity.com/paper/2212.00773/) (2022)
- [Deepfake in the Metaverse: Security Implications for Virtual Gaming, Meetings, and Offices](https://mlsanity.com/paper/2303.14612/) (2023)
- [AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection](https://mlsanity.com/paper/2310.13103/) (2023)
- [Deepfake CAPTCHA: A Method for Preventing Fake Calls](https://mlsanity.com/paper/2301.03064/) (2023)
- [Deepfake Media Forensics: State of the Art and Challenges Ahead](https://mlsanity.com/paper/2408.00388/) (2024)
- [Detecting Audio-Visual Deepfakes with Fine-Grained Inconsistencies](https://mlsanity.com/paper/2408.06753/) (2024)
- [Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges](https://mlsanity.com/paper/2406.06965/) (2024)
- [Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization](https://mlsanity.com/paper/2408.01532/) (2024)
- [CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation](https://mlsanity.com/paper/2505.15233/) (2025)
- [How Good is ChatGPT at Audiovisual Deepfake Detection: A Comparative Study of ChatGPT, AI Models and Human Perception](https://mlsanity.com/paper/2411.09266/) (2024)
- [NPVForensics: Jointing Non-critical Phonemes and Visemes for Deepfake Detection](https://mlsanity.com/paper/2306.06885/) (2023)
- [Exploring the Impact of Moire Pattern on Deepfake Detectors](https://mlsanity.com/paper/2407.10399/) (2024)
- [Audio-Visual Deepfake Detection With Local Temporal Inconsistencies](https://mlsanity.com/paper/2501.08137/) (2025)
- [LLMs Are Not Yet Ready for Deepfake Image Detection](https://mlsanity.com/paper/2506.10474/) (2025)
- [A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection](https://mlsanity.com/paper/2406.14176/) (2024)
- [Statistics-aware Audio-visual Deepfake Detector](https://mlsanity.com/paper/2407.11650/) (2024)
- [AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Deepfake Detection of Frontal Face Videos](https://mlsanity.com/paper/2311.02733/) (2023)
- [WWW: Where, Which and Whatever Enhancing Interpretability in Multimodal Deepfake Detection](https://mlsanity.com/paper/2408.02954/) (2024)
- [From Prediction to Explanation: Multimodal, Explainable, and Interactive Deepfake Detection Framework for Non-Expert Users](https://mlsanity.com/paper/2508.07596/) (2025)
- [Seeing Through the Blur: Unlocking Defocus Maps for Deepfake Detection](https://mlsanity.com/paper/2509.23289/) (2025)
- [Deepfake Detection for Facial Images with Facemasks](https://mlsanity.com/paper/2202.11359/) (2022)
- [Through the Lens: Benchmarking Deepfake Detectors Against Moir'e-Induced Distortions](https://mlsanity.com/paper/2510.23225/) (2025)
- [ExposeAnyone: Personalized Audio-to-Expression Diffusion Models Are Robust Zero-Shot Face Forgery Detectors](https://mlsanity.com/paper/2601.02359/) (2026)

## Abstract (arXiv)

Significant advancements made in the generation of deepfakes have caused security and privacy issues. Attackers can easily impersonate a person's identity in an image by replacing his face with the target person's face. Moreover, a new domain of cloning human voices using deep-learning technologies is also emerging. Now, an attacker can generate realistic cloned voices of humans using only a few seconds of audio of the target person. With the emerging threat of potential harm deepfakes can cause, researchers have proposed deepfake detection methods. However, they only focus on detecting a single modality, i.e., either video or audio. On the other hand, to develop a good deepfake detector that can cope with the recent advancements in deepfake generation, we need to have a detector that can detect deepfakes of multiple modalities, i.e., videos and audios. To build such a detector, we need a dataset that contains video and respective audio deepfakes. We were able to find a most recent deepfake dataset, Audio-Video Multimodal Deepfake Detection Dataset (FakeAVCeleb), that contains not only deepfake videos but synthesized fake audios as well. We used this multimodal deepfake dataset and performed detailed baseline experiments using state-of-the-art unimodal, ensemble-based, and multimodal detection methods to evaluate it. We conclude through detailed experimentation that unimodals, addressing only a single modality, video or audio, do not perform well compared to ensemble-based methods. Whereas purely multimodal-based baselines provide the worst performance.