# Deepfake Caricatures: Amplifying attention to artifacts increases deepfake detection by humans and machines

arXiv: https://arxiv.org/abs/2206.00535 · HTML: https://mlsanity.com/paper/2206.00535/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 60 references that are in the map.
- First posted on arXiv 1 Jun 2022.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [Multi-attentional Deepfake Detection](https://mlsanity.com/paper/2103.02406/) (2021)
- [Detecting Deepfakes with Self-Blended Images](https://mlsanity.com/paper/2204.08376/) (2022)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [Exposing DeepFake Videos By Detecting Face Warping Artifacts](https://mlsanity.com/paper/1811.00656/) (2018)
- [A Short Note on the Kinetics-700 Human Action Dataset](https://mlsanity.com/paper/1907.06987/) (2019)
- [Exposing Deep Fakes Using Inconsistent Head Poses](https://mlsanity.com/paper/1811.00661/) (2018)
- [Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection](https://mlsanity.com/paper/2012.07657/) (2020)
- [On the Detection of Digital Face Manipulation](https://mlsanity.com/paper/1910.01717/) (2019)
- [FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping](https://mlsanity.com/paper/1912.13457/) (2019)
- [The Deepfake Detection Challenge (DFDC) Preview Dataset](https://mlsanity.com/paper/1910.08854/) (2019)
- [Capsule-Forensics: Using Capsule Networks to Detect Forged Images and Videos](https://mlsanity.com/paper/1810.11215/) (2018)
- [Two-branch Recurrent Network for Isolating Deepfakes in Videos](https://mlsanity.com/paper/2008.03412/) (2020)
- [Exploring Temporal Coherence for More General Video Face Forgery Detection](https://mlsanity.com/paper/2108.06693/) (2021)
- [FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals](https://mlsanity.com/paper/1901.02212/) (2019)
- [Mish: A Self Regularized Non-Monotonic Activation Function](https://mlsanity.com/paper/1908.08681/) (2019)
- [Multi-task Learning For Detecting and Segmenting Manipulated Facial Images and Videos](https://mlsanity.com/paper/1906.06876/) (2019)
- [Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection](https://mlsanity.com/paper/2201.07131/) (2022)
- [Recurrent Convolutional Strategies for Face Manipulation Detection in Videos](https://mlsanity.com/paper/1905.00582/) (2019)

## Abstract (arXiv)

Deepfakes can fuel online misinformation. As deepfakes get harder to recognize with the naked eye, human users become more reliant on deepfake detection models to help them decide whether a video is real or fake. Currently, models yield a prediction for a video's authenticity, but do not integrate a method for alerting a human user. We introduce a framework for amplifying artifacts in deepfake videos to make them more detectable by people. We propose a novel, semi-supervised Artifact Attention module, which is trained on human responses to create attention maps that highlight video artifacts, and magnify them to create a novel visual indicator we call "Deepfake Caricatures". In a user study, we demonstrate that Caricatures greatly increase human detection, across video presentation times and user engagement levels. We also introduce a deepfake detection model that incorporates the Artifact Attention module to increase its accuracy and robustness. Overall, we demonstrate the success of a human-centered approach to designing deepfake mitigation methods.