# SAP-DIFF: Semantic Adversarial Patch Generation for Black-Box Face Recognition Models via Diffusion Models

arXiv: https://arxiv.org/abs/2502.19710 · HTML: https://mlsanity.com/paper/2502.19710/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 40 of its 61 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 30 of 40)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Delving into Transferable Adversarial Examples and Black-box Attacks](https://mlsanity.com/paper/1611.02770/) (2016)
- [Adversarial Patch](https://mlsanity.com/paper/1712.09665/) (2017)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Efficient Decision-based Black-box Adversarial Attacks on Face Recognition](https://mlsanity.com/paper/1904.04433/) (2019)
- [Adv-Makeup: A New Imperceptible and Transferable Attack on Face Recognition](https://mlsanity.com/paper/2105.03162/) (2021)
- [Diffusion Models for Imperceptible and Transferable Adversarial Attack](https://mlsanity.com/paper/2305.08192/) (2023)
- [Adversarial Sticker: A Stealthy Attack Method in the Physical World](https://mlsanity.com/paper/2104.06728/) (2021)
- [A General Framework for Adversarial Examples with Objectives](https://mlsanity.com/paper/1801.00349/) (2017)
- [Towards Transferable Adversarial Attack against Deep Face Recognition](https://mlsanity.com/paper/2004.05790/) (2020)
- [Improving Transferability of Adversarial Patches on Face Recognition with Generative Models](https://mlsanity.com/paper/2106.15058/) (2021)
- [Physical Adversarial Attack meets Computer Vision: A Decade Survey](https://mlsanity.com/paper/2209.15179/) (2022)
- [Adv-Attribute: Inconspicuous and Transferable Adversarial Attack on Face Recognition](https://mlsanity.com/paper/2210.06871/) (2022)
- [Simultaneously Optimizing Perturbations and Positions for Black-box Adversarial Patch Attacks](https://mlsanity.com/paper/2212.12995/) (2022)
- [Attacks on State-of-the-Art Face Recognition using Attentional Adversarial Attack Generative Network](https://mlsanity.com/paper/1811.12026/) (2018)
- [Demystifying RCE Vulnerabilities in LLM-Integrated Apps](https://mlsanity.com/paper/2309.02926/) (2023)
- [On adversarial patches: real-world attack on ArcFace-100 face recognition system](https://mlsanity.com/paper/1910.07067/) (2019)
- [Towards Effective Adversarial Textured 3D Meshes on Physical Face Recognition](https://mlsanity.com/paper/2303.15818/) (2023)
- [DiffAM: Diffusion-based Adversarial Makeup Transfer for Facial Privacy Protection](https://mlsanity.com/paper/2405.09882/) (2024)
- [Unravelling Robustness of Deep Learning based Face Recognition Against Adversarial Attacks](https://mlsanity.com/paper/1803.00401/) (2018)
- [Adversarial Learning with Margin-based Triplet Embedding Regularization](https://mlsanity.com/paper/1909.09481/) (2019)
- [Robust Physical-World Attacks on Face Recognition](https://mlsanity.com/paper/2109.09320/) (2021)

## Abstract (arXiv)

Given the need to evaluate the robustness of face recognition (FR) models, many efforts have focused on adversarial patch attacks that mislead FR models by introducing localized perturbations. Impersonation attacks are a significant threat because adversarial perturbations allow attackers to disguise themselves as legitimate users. This can lead to severe consequences, including data breaches, system damage, and misuse of resources. However, research on such attacks in FR remains limited. Existing adversarial patch generation methods exhibit limited efficacy in impersonation attacks due to (1) the need for high attacker capabilities, (2) low attack success rates, and (3) excessive query requirements. To address these challenges, we propose a novel method SAP-DIFF that leverages diffusion models to generate adversarial patches via semantic perturbations in the latent space rather than direct pixel manipulation. We introduce an attention disruption mechanism to generate features unrelated to the original face, facilitating the creation of adversarial samples and a directional loss function to guide perturbations toward the target identity feature space, thereby enhancing attack effectiveness and efficiency. Extensive experiments on popular FR models and datasets demonstrate that our method outperforms state-of-the-art approaches, achieving an average attack success rate improvement of 45.66% (all exceeding 40%), and a reduction in the number of queries by about 40% compared to the SOTA approach