# Improving Transferability of Adversarial Patches on Face Recognition with Generative Models

arXiv: https://arxiv.org/abs/2106.15058 · HTML: https://mlsanity.com/paper/2106.15058/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 29 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)
- [Adversarial Patch](https://mlsanity.com/paper/1712.09665/) (2017)
- [Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks](https://mlsanity.com/paper/1904.02884/) (2019)
- [Generating Adversarial Examples with Adversarial Networks](https://mlsanity.com/paper/1801.02610/) (2018)
- [Generating Natural Adversarial Examples](https://mlsanity.com/paper/1710.11342/) (2017)
- [Efficient Decision-based Black-box Adversarial Attacks on Face Recognition](https://mlsanity.com/paper/1904.04433/) (2019)
- [Physical Adversarial Examples for Object Detectors](https://mlsanity.com/paper/1807.07769/) (2018)
- [Making an Invisibility Cloak: Real World Adversarial Attacks on Object Detectors](https://mlsanity.com/paper/1910.14667/) (2019)
- [AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networks](https://mlsanity.com/paper/1805.11770/) (2018)
- [Constructing Unrestricted Adversarial Examples with Generative Models](https://mlsanity.com/paper/1805.07894/) (2018)
- [SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing](https://mlsanity.com/paper/1906.07927/) (2019)
- [PatchAttack: A Black-box Texture-based Attack with Reinforcement Learning](https://mlsanity.com/paper/2004.05682/) (2020)

## Built on by (most cited first, 30 of 80)

- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [Protecting Facial Privacy: Generating Adversarial Identity Masks via Style-robust Makeup Transfer](https://mlsanity.com/paper/2203.03121/) (2022)
- [Physical Adversarial Attack meets Computer Vision: A Decade Survey](https://mlsanity.com/paper/2209.15179/) (2022)
- [A Survey on Transferability of Adversarial Examples across Deep Neural Networks](https://mlsanity.com/paper/2310.17626/) (2023)
- [Adv-Attribute: Inconspicuous and Transferable Adversarial Attack on Face Recognition](https://mlsanity.com/paper/2210.06871/) (2022)
- [CLIP2Protect: Protecting Facial Privacy using Text-Guided Makeup via Adversarial Latent Search](https://mlsanity.com/paper/2306.10008/) (2023)
- [Simultaneously Optimizing Perturbations and Positions for Black-box Adversarial Patch Attacks](https://mlsanity.com/paper/2212.12995/) (2022)
- [SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation](https://mlsanity.com/paper/2312.04913/) (2023)
- [Detecting and Recovering Sequential DeepFake Manipulation](https://mlsanity.com/paper/2207.02204/) (2022)
- [Sibling-Attack: Rethinking Transferable Adversarial Attacks against Face Recognition](https://mlsanity.com/paper/2303.12512/) (2023)
- [Towards Effective Adversarial Textured 3D Meshes on Physical Face Recognition](https://mlsanity.com/paper/2303.15818/) (2023)
- [DiffAM: Diffusion-based Adversarial Makeup Transfer for Facial Privacy Protection](https://mlsanity.com/paper/2405.09882/) (2024)
- [ImageNet-Patch: A Dataset for Benchmarking Machine Learning Robustness against Adversarial Patches](https://mlsanity.com/paper/2203.04412/) (2022)
- [Robust Physical-World Attacks on Face Recognition](https://mlsanity.com/paper/2109.09320/) (2021)
- [A Survey on Physical Adversarial Attack in Computer Vision](https://mlsanity.com/paper/2209.14262/) (2022)
- [Physical-World Optical Adversarial Attacks on 3D Face Recognition](https://mlsanity.com/paper/2205.13412/) (2022)
- [Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model](https://mlsanity.com/paper/2312.11285/) (2023)
- [Physical Adversarial Attacks For Camera-based Smart Systems: Current Trends, Categorization, Applications, Research Challenges, and Future Outlook](https://mlsanity.com/paper/2308.06173/) (2023)
- [Query-Efficient Decision-based Black-Box Patch Attack](https://mlsanity.com/paper/2307.00477/) (2023)
- [A Survey of Robustness and Safety of 2D and 3D Deep Learning Models Against Adversarial Attacks](https://mlsanity.com/paper/2310.00633/) (2023)
- [Adversarial Relighting Against Face Recognition](https://mlsanity.com/paper/2108.07920/) (2021)
- [DIFFender: Diffusion-Based Adversarial Defense against Patch Attacks](https://mlsanity.com/paper/2306.09124/) (2023)
- [Discrete Point-wise Attack Is Not Enough: Generalized Manifold Adversarial Attack for Face Recognition](https://mlsanity.com/paper/2301.06083/) (2022)
- [Adv-CPG: A Customized Portrait Generation Framework with Facial Adversarial Attacks](https://mlsanity.com/paper/2503.08269/) (2025)
- [Learning to Learn Transferable Attack](https://mlsanity.com/paper/2112.06658/) (2021)
- [Overload: Latency Attacks on Object Detection for Edge Devices](https://mlsanity.com/paper/2304.05370/) (2023)
- [Reliable Evaluation of Adversarial Transferability](https://mlsanity.com/paper/2306.08565/) (2023)
- [When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Models](https://mlsanity.com/paper/2511.21192/) (2025)

## Abstract (arXiv)

Face recognition is greatly improved by deep convolutional neural networks (CNNs). Recently, these face recognition models have been used for identity authentication in security sensitive applications. However, deep CNNs are vulnerable to adversarial patches, which are physically realizable and stealthy, raising new security concerns on the real-world applications of these models. In this paper, we evaluate the robustness of face recognition models using adversarial patches based on transferability, where the attacker has limited accessibility to the target models. First, we extend the existing transfer-based attack techniques to generate transferable adversarial patches. However, we observe that the transferability is sensitive to initialization and degrades when the perturbation magnitude is large, indicating the overfitting to the substitute models. Second, we propose to regularize the adversarial patches on the low dimensional data manifold. The manifold is represented by generative models pre-trained on legitimate human face images. Using face-like features as adversarial perturbations through optimization on the manifold, we show that the gaps between the responses of substitute models and the target models dramatically decrease, exhibiting a better transferability. Extensive digital world experiments are conducted to demonstrate the superiority of the proposed method in the black-box setting. We apply the proposed method in the physical world as well.