# CLIP2Protect: Protecting Facial Privacy using Text-Guided Makeup via Adversarial Latent Search

arXiv: https://arxiv.org/abs/2306.10008 · HTML: https://mlsanity.com/paper/2306.10008/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 79 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 30 of 56)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks](https://mlsanity.com/paper/1904.02884/) (2019)
- [Pivotal Tuning for Latent-based Editing of Real Images](https://mlsanity.com/paper/2106.05744/) (2021)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [Deep Face Recognition: A Survey](https://mlsanity.com/paper/1804.06655/) (2018)
- [CLIPstyler: Image Style Transfer with a Single Text Condition](https://mlsanity.com/paper/2112.00374/) (2021)
- [Spatially Transformed Adversarial Examples](https://mlsanity.com/paper/1801.02612/) (2018)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices](https://mlsanity.com/paper/1804.07573/) (2018)
- [A Survey of Vision-Language Pre-Trained Models](https://mlsanity.com/paper/2202.10936/) (2022)
- [AdvHat: Real-world adversarial attack on ArcFace Face ID system](https://mlsanity.com/paper/1908.08705/) (2019)
- [Constructing Unrestricted Adversarial Examples with Generative Models](https://mlsanity.com/paper/1805.07894/) (2018)
- [Adv-Makeup: A New Imperceptible and Transferable Attack on Face Recognition](https://mlsanity.com/paper/2105.03162/) (2021)
- [LowKey: Leveraging Adversarial Attacks to Protect Social Media Users from Facial Recognition](https://mlsanity.com/paper/2101.07922/) (2021)
- [A General Framework for Adversarial Examples with Objectives](https://mlsanity.com/paper/1801.00349/) (2017)
- [Unrestricted Adversarial Examples via Semantic Manipulation](https://mlsanity.com/paper/1904.06347/) (2019)

## Built on by (most cited first, 30 of 43)

- [DiffAM: Diffusion-based Adversarial Makeup Transfer for Facial Privacy Protection](https://mlsanity.com/paper/2405.09882/) (2024)
- [Transferable Adversarial Facial Images for Privacy Protection](https://mlsanity.com/paper/2408.01428/) (2024)
- [Instruct2Attack: Language-Guided Semantic Adversarial Attacks](https://mlsanity.com/paper/2311.15551/) (2023)
- [Adv-CPG: A Customized Portrait Generation Framework with Facial Adversarial Attacks](https://mlsanity.com/paper/2503.08269/) (2025)
- [First-Place Solution to NeurIPS 2024 Invisible Watermark Removal Challenge](https://mlsanity.com/paper/2508.21072/) (2025)
- [Rethinking Impersonation and Dodging Attacks on Face Recognition Systems](https://mlsanity.com/paper/2401.08903/) (2024)
- [TUNI: A Textual Unimodal Detector for Identity Inference in CLIP Models](https://mlsanity.com/paper/2405.14517/) (2024)
- [Makeup-Guided Facial Privacy Protection via Untrained Neural Network Priors](https://mlsanity.com/paper/2408.12387/) (2024)
- [Enhancing Facial Privacy Protection via Weakening Diffusion Purification](https://mlsanity.com/paper/2503.10350/) (2025)
- [FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing](https://mlsanity.com/paper/2412.19009/) (2024)
- [My Art My Choice: Adversarial Protection Against Unruly AI](https://mlsanity.com/paper/2309.03198/) (2023)
- [Diffusion-based Adversarial Identity Manipulation for Facial Privacy Protection](https://mlsanity.com/paper/2504.21646/) (2025)
- [Adversarial Attacks on Both Face Recognition and Face Anti-spoofing Models](https://mlsanity.com/paper/2405.16940/) (2024)
- [BeautyBank: Encoding Facial Makeup in Latent Space](https://mlsanity.com/paper/2411.11231/) (2024)
- [Improving the Transferability of Adversarial Attacks on Face Recognition with Diverse Parameters Augmentation](https://mlsanity.com/paper/2411.15555/) (2024)
- [Multi-modal Generation via Cross-Modal In-Context Learning](https://mlsanity.com/paper/2405.18304/) (2024)
- [DivTrackee versus DynTracker: Promoting Diversity in Anti-Facial Recognition against Dynamic FR Strategy](https://mlsanity.com/paper/2501.06533/) (2025)
- [Non-Adaptive Adversarial Face Generation](https://mlsanity.com/paper/2507.12107/) (2025)
- [Adversarial Attacks for Good: A Survey of Proactive Protection across the Visual Content Lifecycle](https://mlsanity.com/paper/2608.04314/) (2026)
- [SoK: Analyzing Adversarial Examples: A Framework to Study Adversary Knowledge](https://mlsanity.com/paper/2402.14937/) (2024)
- [ErasableMask: A Robust and Erasable Privacy Protection Scheme against Black-box Face Recognition Models](https://mlsanity.com/paper/2412.17038/) (2024)
- [FaceAnonyMixer: Cancelable Faces via Identity Consistent Latent Space Mixing](https://mlsanity.com/paper/2508.05636/) (2025)
- [Off-The-Shelf Image-to-Image Models Are All You Need To Defeat Image Protection Schemes](https://mlsanity.com/paper/2602.22197/) (2026)
- [Machine Pareidolia: Protecting Facial Image with Emotional Editing](https://mlsanity.com/paper/2603.03665/) (2026)
- [FDeID-Toolbox: Face De-Identification Toolbox](https://mlsanity.com/paper/2603.13121/) (2026)
- [Adv-TGD: Adversarial Text-Guided Diffusion for Face Recognition Impersonation Attacks](https://mlsanity.com/paper/2606.11615/) (2026)
- [Flux-Guard: Facial Identity Protection using diffusion models](https://mlsanity.com/paper/2606.17606/) (2026)
- [Controllable Adversarial Makeup for Privacy via Text-Guided Diffusion](https://mlsanity.com/paper/2503.10549/) (2025)
- [Computation-Efficient and Recognition-Friendly 3D Point Cloud Privacy Protection](https://mlsanity.com/paper/2503.15818/) (2025)
- [DiffClean: Diffusion-based Makeup Removal for Accurate Age Estimation](https://mlsanity.com/paper/2507.13292/) (2025)

## Abstract (arXiv)

The success of deep learning based face recognition systems has given rise to serious privacy concerns due to their ability to enable unauthorized tracking of users in the digital world. Existing methods for enhancing privacy fail to generate naturalistic images that can protect facial privacy without compromising user experience. We propose a novel two-step approach for facial privacy protection that relies on finding adversarial latent codes in the low-dimensional manifold of a pretrained generative model. The first step inverts the given face image into the latent space and finetunes the generative model to achieve an accurate reconstruction of the given image from its latent code. This step produces a good initialization, aiding the generation of high-quality faces that resemble the given identity. Subsequently, user-defined makeup text prompts and identity-preserving regularization are used to guide the search for adversarial codes in the latent space. Extensive experiments demonstrate that faces generated by our approach have stronger black-box transferability with an absolute gain of 12.06% over the state-of-the-art facial privacy protection approach under the face verification task. Finally, we demonstrate the effectiveness of the proposed approach for commercial face recognition systems. Our code is available at https://github.com/fahadshamshad/Clip2Protect.