# Detection and Localization of Facial Expression Manipulations

arXiv: https://arxiv.org/abs/2103.08134 · HTML: https://mlsanity.com/paper/2103.08134/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 67 references that are in the map.
- First posted on arXiv 15 Mar 2021.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- [Exposing DeepFake Videos By Detecting Face Warping Artifacts](https://mlsanity.com/paper/1811.00656/) (2018)
- [Exposing Deep Fakes Using Inconsistent Head Poses](https://mlsanity.com/paper/1811.00661/) (2018)
- [On the Detection of Digital Face Manipulation](https://mlsanity.com/paper/1910.01717/) (2019)
- [DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection](https://mlsanity.com/paper/2001.03024/) (2020)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [DeepFakes: a New Threat to Face Recognition? Assessment and Detection](https://mlsanity.com/paper/1812.08685/) (2018)
- [The Deepfake Detection Challenge (DFDC) Preview Dataset](https://mlsanity.com/paper/1910.08854/) (2019)
- [Capsule-Forensics: Using Capsule Networks to Detect Forged Images and Videos](https://mlsanity.com/paper/1810.11215/) (2018)
- [Object Detectors Emerge in Deep Scene CNNs](https://mlsanity.com/paper/1412.6856/) (2014)
- [Learning Rich Features for Image Manipulation Detection](https://mlsanity.com/paper/1805.04953/) (2018)
- [The DeepFake Detection Challenge (DFDC) Dataset](https://mlsanity.com/paper/2006.07397/) (2020)
- [Multi-task Learning For Detecting and Segmenting Manipulated Facial Images and Videos](https://mlsanity.com/paper/1906.06876/) (2019)
- [Two-Stream Neural Networks for Tampered Face Detection](https://mlsanity.com/paper/1803.11276/) (2018)
- [FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces](https://mlsanity.com/paper/1803.09179/) (2018)
- [Recurrent Convolutional Strategies for Face Manipulation Detection in Videos](https://mlsanity.com/paper/1905.00582/) (2019)
- [ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection](https://mlsanity.com/paper/1812.02510/) (2018)
- [Hybrid LSTM and Encoder-Decoder Architecture for Detection of Image Forgeries](https://mlsanity.com/paper/1903.02495/) (2019)

## Built on by (most cited first, 15 of 15)

- [LEGION: Learning to Ground and Explain for Synthetic Image Detection](https://mlsanity.com/paper/2503.15264/) (2025)
- [Towards a Universal Synthetic Video Detector: From Face or Background Manipulations to Fully AI-Generated Content](https://mlsanity.com/paper/2412.12278/) (2024)
- [Digital and Physical Face Attacks: Reviewing and One Step Further](https://mlsanity.com/paper/2209.14692/) (2022)
- [EKILA: Synthetic Media Provenance and Attribution for Generative Art](https://mlsanity.com/paper/2304.04639/) (2023)
- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [Causality-Inspired Taxonomy for Explainable Artificial Intelligence](https://mlsanity.com/paper/2208.09500/) (2022)
- [GOTCHA: Real-Time Video Deepfake Detection via Challenge-Response](https://mlsanity.com/paper/2210.06186/) (2022)
- [VADER: Video Alignment Differencing and Retrieval](https://mlsanity.com/paper/2303.13193/) (2023)
- [Open Set Classification of GAN-based Image Manipulations via a ViT-based Hybrid Architecture](https://mlsanity.com/paper/2304.05212/) (2023)
- [Passive Deepfake Detection Across Multi-modalities: A Comprehensive Survey](https://mlsanity.com/paper/2411.17911/) (2024)
- [Automated Deepfake Detection](https://mlsanity.com/paper/2106.10705/) (2021)
- [AI-based Identity Fraud Detection: A Systematic Review](https://mlsanity.com/paper/2501.09239/) (2025)
- [An Architecture for the detection of GAN-generated Flood Images with Localization Capabilities](https://mlsanity.com/paper/2205.07073/) (2022)
- [Towards Generalizable Deepfake Detection by Primary Region Regularization](https://mlsanity.com/paper/2307.12534/) (2023)
- [DeepFidelity: Perceptual Forgery Fidelity Assessment for Deepfake Detection](https://mlsanity.com/paper/2312.04961/) (2023)

## Abstract (arXiv)

Concern regarding the wide-spread use of fraudulent images/videos in social media necessitates precise detection of such fraud. The importance of facial expressions in communication is widely known, and adversarial attacks often focus on manipulating the expression related features. Thus, it is important to develop methods that can detect manipulations in facial expressions, and localize the manipulated regions. To address this problem, we propose a framework that is able to detect manipulations in facial expression using a close combination of facial expression recognition and image manipulation methods. With the addition of feature maps extracted from the facial expression recognition framework, our manipulation detector is able to localize the manipulated region. We show that, on the Face2Face dataset, where there is abundant expression manipulation, our method achieves over 3% higher accuracy for both classification and localization of manipulations compared to state-of-the-art methods. In addition, results on the NeuralTextures dataset where the facial expressions corresponding to the mouth regions have been modified, show 2% higher accuracy in both classification and localization of manipulation. We demonstrate that the method performs at-par with the state-of-the-art methods in cases where the expression is not manipulated, but rather the identity is changed, thus ensuring generalizability of the approach.