# A Multimodal Framework for Deepfake Detection

arXiv: https://arxiv.org/abs/2410.03487 · HTML: https://mlsanity.com/paper/2410.03487/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 36 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 10 of 10)

- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- [Exposing DeepFake Videos By Detecting Face Warping Artifacts](https://mlsanity.com/paper/1811.00656/) (2018)
- [Exposing Deep Fakes Using Inconsistent Head Poses](https://mlsanity.com/paper/1811.00661/) (2018)
- [DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection](https://mlsanity.com/paper/2001.00179/) (2020)
- [DeepFakes: a New Threat to Face Recognition? Assessment and Detection](https://mlsanity.com/paper/1812.08685/) (2018)
- [Deepfakes Detection with Automatic Face Weighting](https://mlsanity.com/paper/2004.12027/) (2020)
- [Deepfake Detection using Spatiotemporal Convolutional Networks](https://mlsanity.com/paper/2006.14749/) (2020)
- [Deepfake Videos in the Wild: Analysis and Detection](https://mlsanity.com/paper/2103.04263/) (2021)
- [GenConViT: Deepfake Video Detection Using Generative Convolutional Vision Transformer](https://mlsanity.com/paper/2307.07036/) (2023)
- [DeepFake Detection with Inconsistent Head Poses: Reproducibility and Analysis](https://mlsanity.com/paper/2108.12715/) (2021)

## Built on by (most cited first, 3 of 3)

- [Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions](https://mlsanity.com/paper/2507.02900/) (2025)
- [A Multi-Level Strategy for Deepfake Content Moderation under EU Regulation](https://mlsanity.com/paper/2507.08879/) (2025)
- [Deepfake Detection in Social Media: A Temporal Artifact Analysis Using 3D Convolutional Neural Networks](https://mlsanity.com/paper/2605.17573/) (2026)

## Abstract (arXiv)

The rapid advancement of deepfake technology poses a significant threat to digital media integrity. Deepfakes, synthetic media created using AI, can convincingly alter videos and audio to misrepresent reality. This creates risks of misinformation, fraud, and severe implications for personal privacy and security. Our research addresses the critical issue of deepfakes through an innovative multimodal approach, targeting both visual and auditory elements. This comprehensive strategy recognizes that human perception integrates multiple sensory inputs, particularly visual and auditory information, to form a complete understanding of media content. For visual analysis, a model that employs advanced feature extraction techniques was developed, extracting nine distinct facial characteristics and then applying various machine learning and deep learning models. For auditory analysis, our model leverages mel-spectrogram analysis for feature extraction and then applies various machine learning and deep learningmodels. To achieve a combined analysis, real and deepfake audio in the original dataset were swapped for testing purposes and ensured balanced samples. Using our proposed models for video and audio classification i.e. Artificial Neural Network and VGG19, the overall sample is classified as deepfake if either component is identified as such. Our multimodal framework combines visual and auditory analyses, yielding an accuracy of 94%.