# Watch Those Words: Video Falsification Detection Using Word-Conditioned Facial Motion

arXiv: https://arxiv.org/abs/2112.10936 · HTML: https://mlsanity.com/paper/2112.10936/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 54 references that are in the map.
- First posted on arXiv 21 Dec 2021.

## Builds on (most cited first, 27 of 27)

- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [CNN-generated images are surprisingly easy to spot... for now](https://mlsanity.com/paper/1912.11035/) (2019)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues](https://mlsanity.com/paper/2007.09355/) (2020)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Multi-attentional Deepfake Detection](https://mlsanity.com/paper/2103.02406/) (2021)
- [FSGAN: Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/1908.05932/) (2019)
- [Exposing DeepFake Videos By Detecting Face Warping Artifacts](https://mlsanity.com/paper/1811.00656/) (2018)
- [Exposing Deep Fakes Using Inconsistent Head Poses](https://mlsanity.com/paper/1811.00661/) (2018)
- [Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection](https://mlsanity.com/paper/2012.07657/) (2020)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Neural Voice Puppetry: Audio-driven Facial Reenactment](https://mlsanity.com/paper/1912.05566/) (2019)
- [The Deepfake Detection Challenge (DFDC) Preview Dataset](https://mlsanity.com/paper/1910.08854/) (2019)
- [Learning Individual Styles of Conversational Gesture](https://mlsanity.com/paper/1906.04160/) (2019)
- [Capsule-Forensics: Using Capsule Networks to Detect Forged Images and Videos](https://mlsanity.com/paper/1810.11215/) (2018)
- [FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals](https://mlsanity.com/paper/1901.02212/) (2019)
- [Two-Stream Neural Networks for Tampered Face Detection](https://mlsanity.com/paper/1803.11276/) (2018)
- [M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection](https://mlsanity.com/paper/2104.09770/) (2021)
- [Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using Affective Cues](https://mlsanity.com/paper/2003.06711/) (2020)
- [Text-based Editing of Talking-head Video](https://mlsanity.com/paper/1906.01524/) (2019)
- [ID-Reveal: Identity-aware DeepFake Video Detection](https://mlsanity.com/paper/2012.02512/) (2020)
- [Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization](https://mlsanity.com/paper/2005.14405/) (2020)
- [Evading Deepfake-Image Detectors with White- and Black-Box Attacks](https://mlsanity.com/paper/2004.00622/) (2020)
- [Detecting Deep-Fake Videos from Appearance and Behavior](https://mlsanity.com/paper/2004.14491/) (2020)
- [DeepRhythm: Exposing DeepFakes with Attentional Visual Heartbeat Rhythms](https://mlsanity.com/paper/2006.07634/) (2020)

## Built on by (most cited first, 15 of 15)

- [Audio-Visual Person-of-Interest DeepFake Detection](https://mlsanity.com/paper/2204.03083/) (2022)
- [Human Detection of Political Speech Deepfakes across Transcripts, Audio, and Video](https://mlsanity.com/paper/2202.12883/) (2022)
- [Exposing Lip-syncing Deepfakes from Mouth Inconsistencies](https://mlsanity.com/paper/2401.10113/) (2024)
- [Avatar Fingerprinting for Authorized Use of Synthetic Talking-Head Videos](https://mlsanity.com/paper/2305.03713/) (2023)
- [Deepfake Media Forensics: State of the Art and Challenges Ahead](https://mlsanity.com/paper/2408.00388/) (2024)
- [Detecting Audio-Visual Deepfakes with Fine-Grained Inconsistencies](https://mlsanity.com/paper/2408.06753/) (2024)
- [Detecting Lip-Syncing Deepfakes: Vision Temporal Transformer for Analyzing Mouth Inconsistencies](https://mlsanity.com/paper/2504.01470/) (2025)
- [Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization](https://mlsanity.com/paper/2408.01532/) (2024)
- [Passive Deepfake Detection Across Multi-modalities: A Comprehensive Survey](https://mlsanity.com/paper/2411.17911/) (2024)
- [Statistics-aware Audio-visual Deepfake Detector](https://mlsanity.com/paper/2407.11650/) (2024)
- [Protecting President Zelenskyy against Deep Fakes](https://mlsanity.com/paper/2206.12043/) (2022)
- [Alignment Helps Make the Most of Multimodal Data](https://mlsanity.com/paper/2405.08454/) (2024)
- [Combating Falsification of Speech Videos with Live Optical Signatures (Extended Version)](https://mlsanity.com/paper/2504.21846/) (2025)
- [ExposeAnyone: Personalized Audio-to-Expression Diffusion Models Are Robust Zero-Shot Face Forgery Detectors](https://mlsanity.com/paper/2601.02359/) (2026)
- [CUPID: Reconstructing UV Texture Maps for Interpretable Person-of-Interest Deepfake Detection](https://mlsanity.com/paper/2606.20302/) (2026)

## Abstract (arXiv)

In today's era of digital misinformation, we are increasingly faced with new threats posed by video falsification techniques. Such falsifications range from cheapfakes (e.g., lookalikes or audio dubbing) to deepfakes (e.g., sophisticated AI media synthesis methods), which are becoming perceptually indistinguishable from real videos. To tackle this challenge, we propose a multi-modal semantic forensic approach to discover clues that go beyond detecting discrepancies in visual quality, thereby handling both simpler cheapfakes and visually persuasive deepfakes. In this work, our goal is to verify that the purported person seen in the video is indeed themselves by detecting anomalous facial movements corresponding to the spoken words. We leverage the idea of attribution to learn person-specific biometric patterns that distinguish a given speaker from others. We use interpretable Action Units (AUs) to capture a person's face and head movement as opposed to deep CNN features, and we are the first to use word-conditioned facial motion analysis. We further demonstrate our method's effectiveness on a range of fakes not seen in training including those without video manipulation, that were not addressed in prior work.