# Detecting Audio-Visual Deepfakes with Fine-Grained Inconsistencies

arXiv: https://arxiv.org/abs/2408.06753 · HTML: https://mlsanity.com/paper/2408.06753/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 13 Aug 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- [Multi-attentional Deepfake Detection](https://mlsanity.com/paper/2103.02406/) (2021)
- [Detecting Deepfakes with Self-Blended Images](https://mlsanity.com/paper/2204.08376/) (2022)
- [FSGAN: Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/1908.05932/) (2019)
- [Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection](https://mlsanity.com/paper/2012.07657/) (2020)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks](https://mlsanity.com/paper/2110.01200/) (2021)
- [Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection](https://mlsanity.com/paper/2203.12208/) (2022)
- [End-to-end anti-spoofing with RawNet2](https://mlsanity.com/paper/2011.01108/) (2020)
- [Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection](https://mlsanity.com/paper/2201.07131/) (2022)
- [Fast Face-swap Using Convolutional Neural Networks](https://mlsanity.com/paper/1611.09577/) (2016)
- [Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using Affective Cues](https://mlsanity.com/paper/2003.06711/) (2020)
- [Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization](https://mlsanity.com/paper/2005.14405/) (2020)
- [Self-Supervised Video Forensics by Audio-Visual Anomaly Detection](https://mlsanity.com/paper/2301.01767/) (2023)
- [Deepfake Video Detection Using Convolutional Vision Transformer](https://mlsanity.com/paper/2102.11126/) (2021)
- [AltFreezing for More General Video Face Forgery Detection](https://mlsanity.com/paper/2307.08317/) (2023)
- [Voice-Face Homogeneity Tells Deepfake](https://mlsanity.com/paper/2203.02195/) (2022)
- [Audio-Visual Person-of-Interest DeepFake Detection](https://mlsanity.com/paper/2204.03083/) (2022)
- [RawNet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification](https://mlsanity.com/paper/1904.08104/) (2019)
- [Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors](https://mlsanity.com/paper/2109.02993/) (2021)
- [Watch Those Words: Video Falsification Detection Using Word-Conditioned Facial Motion](https://mlsanity.com/paper/2112.10936/) (2021)
- [AVT2-DWF: Improving Deepfake Detection with Audio-Visual Fusion and Dynamic Weighting Strategies](https://mlsanity.com/paper/2403.14974/) (2024)

## Built on by (most cited first, 7 of 7)

- [Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024](https://mlsanity.com/paper/2503.02857/) (2025)
- [FakeFormer: Efficient Vulnerability-Driven Transformers for Generalisable Deepfake Detection](https://mlsanity.com/paper/2410.21964/) (2024)
- [Audio-Visual Deepfake Detection With Local Temporal Inconsistencies](https://mlsanity.com/paper/2501.08137/) (2025)
- [Next-Frame Feature Prediction for Multimodal Deepfake Detection and Temporal Localization](https://mlsanity.com/paper/2511.10212/) (2025)
- [Are DeepFakes Realistic Enough? Exploring Semantic Mismatch as a Novel Challenge](https://mlsanity.com/paper/2604.28022/) (2026)
- [PIVOT: Physics-Grounded Verification for AI-Generated Audio-Video Detection](https://mlsanity.com/paper/2609.15562/) (2026)
- [Physics-Aware Deepfake Detection via Distance-Speech Consistency](https://mlsanity.com/paper/2610.18485/) (2026)

## Abstract (arXiv)

Existing methods on audio-visual deepfake detection mainly focus on high-level features for modeling inconsistencies between audio and visual data. As a result, these approaches usually overlook finer audio-visual artifacts, which are inherent to deepfakes. Herein, we propose the introduction of fine-grained mechanisms for detecting subtle artifacts in both spatial and temporal domains. First, we introduce a local audio-visual model capable of capturing small spatial regions that are prone to inconsistencies with audio. For that purpose, a fine-grained mechanism based on a spatially-local distance coupled with an attention module is adopted. Second, we introduce a temporally-local pseudo-fake augmentation to include samples incorporating subtle temporal inconsistencies in our training set. Experiments on the DFDC and the FakeAVCeleb datasets demonstrate the superiority of the proposed method in terms of generalization as compared to the state-of-the-art under both in-dataset and cross-dataset settings.