# AOT: Appearance Optimal Transport Based Identity Swapping for Forgery Detection

arXiv: https://arxiv.org/abs/2011.02674 · HTML: https://mlsanity.com/paper/2011.02674/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2020.

## Built on by (most cited first, 16 of 16)

- [Generalizing Face Forgery Detection with High-frequency Features](https://mlsanity.com/paper/2103.12376/) (2021)
- [CelebV-HQ: A Large-Scale Video Facial Attributes Dataset](https://mlsanity.com/paper/2207.12393/) (2022)
- [HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping](https://mlsanity.com/paper/2106.09965/) (2021)
- [Countering Malicious DeepFakes: Survey, Battleground, and Horizon](https://mlsanity.com/paper/2103.00218/) (2021)
- [Delving into Sequential Patches for Deepfake Detection](https://mlsanity.com/paper/2207.02803/) (2022)
- [StyleSwap: Style-Based Generator Empowers Robust Face Swapping](https://mlsanity.com/paper/2209.13514/) (2022)
- [Inverting Generative Adversarial Renderer for Face Reconstruction](https://mlsanity.com/paper/2105.02431/) (2021)
- [Learning Disentangled Representation for One-shot Progressive Face Swapping](https://mlsanity.com/paper/2203.12985/) (2022)
- [ReliableSwap: Boosting General Face Swapping Via Reliable Supervision](https://mlsanity.com/paper/2306.05356/) (2023)
- [E4S: Fine-grained Face Swapping via Editing With Regional GAN Inversion](https://mlsanity.com/paper/2310.15081/) (2023)
- [FlowFace: Semantic Flow-guided Shape-aware Face Swapping](https://mlsanity.com/paper/2212.02797/) (2022)
- [Heterogeneous Face Recognition via Face Synthesis with Identity-Attribute Disentanglement](https://mlsanity.com/paper/2206.04854/) (2022)
- [Exploring Spatial-Temporal Features for Deepfake Detection and Localization](https://mlsanity.com/paper/2210.15872/) (2022)
- [Causal Representation Learning for Context-Aware Face Transfer](https://mlsanity.com/paper/2110.01571/) (2021)
- [FlowFace++: Explicit Semantic Flow-supervised End-to-End Face Swapping](https://mlsanity.com/paper/2306.12686/) (2023)
- [Dual Defense: Adversarial, Traceable, and Invisible Robust Watermarking against Face Swapping](https://mlsanity.com/paper/2310.16540/) (2023)

## Abstract (arXiv)

Recent studies have shown that the performance of forgery detection can be improved with diverse and challenging Deepfakes datasets. However, due to the lack of Deepfakes datasets with large variance in appearance, which can be hardly produced by recent identity swapping methods, the detection algorithm may fail in this situation. In this work, we provide a new identity swapping algorithm with large differences in appearance for face forgery detection. The appearance gaps mainly arise from the large discrepancies in illuminations and skin colors that widely exist in real-world scenarios. However, due to the difficulties of modeling the complex appearance mapping, it is challenging to transfer fine-grained appearances adaptively while preserving identity traits. This paper formulates appearance mapping as an optimal transport problem and proposes an Appearance Optimal Transport model (AOT) to formulate it in both latent and pixel space. Specifically, a relighting generator is designed to simulate the optimal transport plan. It is solved via minimizing the Wasserstein distance of the learned features in the latent space, enabling better performance and less computation than conventional optimization. To further refine the solution of the optimal transport plan, we develop a segmentation game to minimize the Wasserstein distance in the pixel space. A discriminator is introduced to distinguish the fake parts from a mix of real and fake image patches. Extensive experiments reveal that the superiority of our method when compared with state-of-the-art methods and the ability of our generated data to improve the performance of face forgery detection.