# Detecting Lip-Syncing Deepfakes: Vision Temporal Transformer for Analyzing Mouth Inconsistencies

arXiv: https://arxiv.org/abs/2504.01470 · HTML: https://mlsanity.com/paper/2504.01470/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 62 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [FSGAN: Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/1908.05932/) (2019)
- [Exposing DeepFake Videos By Detecting Face Warping Artifacts](https://mlsanity.com/paper/1811.00656/) (2018)
- [Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection](https://mlsanity.com/paper/2012.07657/) (2020)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [Exploring Temporal Coherence for More General Video Face Forgery Detection](https://mlsanity.com/paper/2108.06693/) (2021)
- [DeepFaceLab: Integrated, flexible and extensible face-swapping framework](https://mlsanity.com/paper/2005.05535/) (2020)
- [Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection](https://mlsanity.com/paper/2201.07131/) (2022)
- [Transcending Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection](https://mlsanity.com/paper/2311.11278/) (2023)
- [Fast Face-swap Using Convolutional Neural Networks](https://mlsanity.com/paper/1611.09577/) (2016)
- [VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild](https://mlsanity.com/paper/2211.14758/) (2022)
- [Deep Learning for Deepfakes Creation and Detection: A Survey](https://mlsanity.com/paper/1909.11573/) (2019)
- [Identity-Preserving Talking Face Generation with Landmark and Appearance Priors](https://mlsanity.com/paper/2305.08293/) (2023)
- [Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose](https://mlsanity.com/paper/2002.10137/) (2020)
- [Self-Supervised Video Forensics by Audio-Visual Anomaly Detection](https://mlsanity.com/paper/2301.01767/) (2023)
- [KoDF: A Large-scale Korean DeepFake Detection Dataset](https://mlsanity.com/paper/2103.10094/) (2021)
- [AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection](https://mlsanity.com/paper/2406.02951/) (2024)
- [DeepFake Detection Based on the Discrepancy Between the Face and its Context](https://mlsanity.com/paper/2008.12262/) (2020)

## Built on by (most cited first, 7 of 7)

- [FauForensics: Boosting Audio-Visual Deepfake Detection with Facial Action Units](https://mlsanity.com/paper/2505.08294/) (2025)
- [Video Forgery Detection with Optical Flow Residuals and Spatial-Temporal Consistency](https://mlsanity.com/paper/2508.00397/) (2025)
- [SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment](https://mlsanity.com/paper/2603.25140/) (2026)
- [BioLip: Language-Generalizable Lip-Sync Deepfake Detection via Biomechanical Constraint Violation Modeling](https://mlsanity.com/paper/2604.16808/) (2026)
- [The Alpha Blending Hypothesis: Compositing Shortcut in Deepfake Detection](https://mlsanity.com/paper/2605.10334/) (2026)
- [Physiological Signals as a Forensic Modality for Talking-Face Deepfake Detection](https://mlsanity.com/paper/2607.21776/) (2026)
- [Complementary rPPG-Derived and Lip-Region Frequency Cues for Talking-Face Deepfake Detection](https://mlsanity.com/paper/2609.22284/) (2026)

## Abstract (arXiv)

Deepfakes are AI-generated media in which the original content is digitally altered to create convincing but manipulated images, videos, or audio. Among the various types of deepfakes, lip-syncing deepfakes are one of the most challenging deepfakes to detect. In these videos, a person's lip movements are synthesized to match altered or entirely new audio using AI models. Therefore, unlike other types of deepfakes, the artifacts in lip-syncing deepfakes are confined to the mouth region, making them more subtle and, thus harder to discern. In this paper, we propose LIPINC-V2, a novel detection framework that leverages a combination of vision temporal transformer with multihead cross-attention to detect lip-syncing deepfakes by identifying spatiotemporal inconsistencies in the mouth region. These inconsistencies appear across adjacent frames and persist throughout the video. Our model can successfully capture both short-term and long-term variations in mouth movement, enhancing its ability to detect these inconsistencies. Additionally, we created a new lip-syncing deepfake dataset, LipSyncTIMIT, which was generated using five state-of-the-art lip-syncing models to simulate real-world scenarios. Extensive experiments on our proposed LipSyncTIMIT dataset and two other benchmark deepfake datasets demonstrate that our model achieves state-of-the-art performance. The code and the dataset are available at https://github.com/skrantidatta/LIPINC-V2 .