# Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes

arXiv: https://arxiv.org/abs/2401.15668 · HTML: https://mlsanity.com/paper/2401.15668/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 54 references that are in the map.
- First posted on arXiv 28 Jan 2024.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Towards Universal Fake Image Detectors that Generalize Across Generative Models](https://mlsanity.com/paper/2302.10174/) (2023)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [Multi-attentional Deepfake Detection](https://mlsanity.com/paper/2103.02406/) (2021)
- [Detecting Deepfakes with Self-Blended Images](https://mlsanity.com/paper/2204.08376/) (2022)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection](https://mlsanity.com/paper/2012.07657/) (2020)
- [What makes fake images detectable? Understanding properties that generalize](https://mlsanity.com/paper/2008.10588/) (2020)
- [Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain](https://mlsanity.com/paper/2103.01856/) (2021)
- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection](https://mlsanity.com/paper/2201.07131/) (2022)
- [STGAN: A Unified Selective Transfer Network for Arbitrary Image Attribute Editing](https://mlsanity.com/paper/1904.09709/) (2019)
- [Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using Affective Cues](https://mlsanity.com/paper/2003.06711/) (2020)
- [Local Relation Learning for Face Forgery Detection](https://mlsanity.com/paper/2105.02577/) (2021)
- [Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert](https://mlsanity.com/paper/2303.17480/) (2023)
- [Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization](https://mlsanity.com/paper/2005.14405/) (2020)
- [StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator](https://mlsanity.com/paper/2305.05445/) (2023)
- [Deepfake Video Detection Using Convolutional Vision Transformer](https://mlsanity.com/paper/2102.11126/) (2021)
- [Countering Malicious DeepFakes: Survey, Battleground, and Horizon](https://mlsanity.com/paper/2103.00218/) (2021)
- [Audio-Visual Person-of-Interest DeepFake Detection](https://mlsanity.com/paper/2204.03083/) (2022)
- [GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation](https://mlsanity.com/paper/2305.00787/) (2023)
- [Anti-Forgery: Towards a Stealthy and Robust DeepFake Disruption Attack via Adversarial Perceptual-aware Perturbations](https://mlsanity.com/paper/2206.00477/) (2022)

## Built on by (most cited first, 30 of 32)

- [Diffusion Model-Based Video Editing: A Survey](https://mlsanity.com/paper/2407.07111/) (2024)
- [Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning](https://mlsanity.com/paper/2412.00175/) (2024)
- [Deepfake Detection that Generalizes Across Benchmarks](https://mlsanity.com/paper/2508.06248/) (2025)
- [TalkingHeadBench: A Multi-Modal Benchmark & Analysis of Talking-Head DeepFake Detection](https://mlsanity.com/paper/2505.24866/) (2025)
- [Unlocking the Hidden Potential of CLIP in Generalizable Deepfake Detection](https://mlsanity.com/paper/2503.19683/) (2025)
- [Detecting Lip-Syncing Deepfakes: Vision Temporal Transformer for Analyzing Mouth Inconsistencies](https://mlsanity.com/paper/2504.01470/) (2025)
- [The DeepSpeak Dataset](https://mlsanity.com/paper/2408.05366/) (2024)
- [CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation](https://mlsanity.com/paper/2505.15233/) (2025)
- [PIA: Deepfake Detection Using Phoneme-Temporal and Identity-Dynamic Analysis](https://mlsanity.com/paper/2510.14241/) (2025)
- [MVAD: A Benchmark Dataset for Multimodal AI-Generated Video-Audio Detection](https://mlsanity.com/paper/2512.00336/) (2025)
- [SocialDF: Benchmark Dataset and Detection Model for Mitigating Harmful Deepfake Content on Social Media Platforms](https://mlsanity.com/paper/2506.05538/) (2025)
- [FauForensics: Boosting Audio-Visual Deepfake Detection with Facial Action Units](https://mlsanity.com/paper/2505.08294/) (2025)
- [DeepFake Doctor: Diagnosing and Treating Audio-Video Fake Detection](https://mlsanity.com/paper/2506.05851/) (2025)
- [Detecting Deepfake Talking Heads from Facial Biometric Anomalies](https://mlsanity.com/paper/2507.08917/) (2025)
- [Investigating self-supervised representations for audio-visual deepfake detection](https://mlsanity.com/paper/2511.17181/) (2025)
- [AVFakeBench: A Comprehensive Audio-Video Forgery Detection Benchmark for AV-LMMs](https://mlsanity.com/paper/2511.21251/) (2025)
- [DeepAgent: A Dual Stream Multi Agent Fusion for Robust Multimodal Deepfake Detection](https://mlsanity.com/paper/2512.07351/) (2025)
- [ExposeAnyone: Personalized Audio-to-Expression Diffusion Models Are Robust Zero-Shot Face Forgery Detectors](https://mlsanity.com/paper/2601.02359/) (2026)
- [Leave No Stone Unturned: Uncovering Holistic Audio-Visual Intrinsic Coherence for Deepfake Detection](https://mlsanity.com/paper/2603.23960/) (2026)
- [Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems](https://mlsanity.com/paper/2507.21157/) (2025)
- [Referee: Reference-aware Audiovisual Deepfake Detection](https://mlsanity.com/paper/2510.27475/) (2025)
- [BioLip: Language-Generalizable Lip-Sync Deepfake Detection via Biomechanical Constraint Violation Modeling](https://mlsanity.com/paper/2604.16808/) (2026)
- [The Alpha Blending Hypothesis: Compositing Shortcut in Deepfake Detection](https://mlsanity.com/paper/2605.10334/) (2026)
- [CAM-VFD: Cross-Attention Multimodal Video Forgery Detection](https://mlsanity.com/paper/2605.17133/) (2026)
- [EMO-BOOST: Emotion-Augmented Audio-Visual Features for Improved Generalization in Deepfake Detection](https://mlsanity.com/paper/2605.19630/) (2026)
- [Inconsistency-aware Multimodal Schr"odinger Bridge for Deepfake Localization](https://mlsanity.com/paper/2605.23113/) (2026)
- [From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection](https://mlsanity.com/paper/2605.27944/) (2026)
- [Physiological Signals as a Forensic Modality for Talking-Face Deepfake Detection](https://mlsanity.com/paper/2607.21776/) (2026)
- [Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AI](https://mlsanity.com/paper/2608.05258/) (2026)
- [Foundation Models are Implicit Deepfake Detectors](https://mlsanity.com/paper/2608.09427/) (2026)

## Abstract (arXiv)

In recent years, DeepFake technology has achieved unprecedented success in high-quality video synthesis, but these methods also pose potential and severe security threats to humanity. DeepFake can be bifurcated into entertainment applications like face swapping and illicit uses such as lip-syncing fraud. However, lip-forgery videos, which neither change identity nor have discernible visual artifacts, present a formidable challenge to existing DeepFake detection methods. Our preliminary experiments have shown that the effectiveness of the existing methods often drastically decrease or even fail when tackling lip-syncing videos. In this paper, for the first time, we propose a novel approach dedicated to lip-forgery identification that exploits the inconsistency between lip movements and audio signals. We also mimic human natural cognition by capturing subtle biological links between lips and head regions to boost accuracy. To better illustrate the effectiveness and advances of our proposed method, we create a high-quality LipSync dataset, AVLips, by employing the state-of-the-art lip generators. We hope this high-quality and diverse dataset could be well served the further research on this challenging and interesting field. Experimental results show that our approach gives an average accuracy of more than 95.3% in spotting lip-syncing videos, significantly outperforming the baselines. Extensive experiments demonstrate the capability to tackle deepfakes and the robustness in surviving diverse input transformations. Our method achieves an accuracy of up to 90.2% in real-world scenarios (e.g., WeChat video call) and shows its powerful capabilities in real scenario deployment. To facilitate the progress of this research community, we release all resources at https://github.com/AaronComo/LipFD.