# Safeguarding Facial Identity against Diffusion-based Face Swapping via Cascading Pathway Disruption

arXiv: https://arxiv.org/abs/2601.14738 · HTML: https://mlsanity.com/paper/2601.14738/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 25 of 25)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [InstantID: Zero-shot Identity-Preserving Generation in Seconds](https://mlsanity.com/paper/2401.07519/) (2024)
- [SimSwap: An Efficient Framework For High Fidelity Face Swapping](https://mlsanity.com/paper/2106.06340/) (2021)
- [Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples](https://mlsanity.com/paper/2302.04578/) (2023)
- [Raising the Cost of Malicious AI-Powered Image Editing](https://mlsanity.com/paper/2302.06588/) (2023)
- [Anti-DreamBooth: Protecting users from personalized text-to-image synthesis](https://mlsanity.com/paper/2303.15433/) (2023)
- [One Shot Face Swapping on Megapixels](https://mlsanity.com/paper/2105.04932/) (2021)
- [Mist: Towards Improved Adversarial Examples for Diffusion Models](https://mlsanity.com/paper/2305.12683/) (2023)
- [HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping](https://mlsanity.com/paper/2106.09965/) (2021)
- [DiffFace: Diffusion-based Face Swapping with Facial Guidance](https://mlsanity.com/paper/2212.13344/) (2022)
- [Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Control](https://mlsanity.com/paper/2405.12970/) (2024)
- [Exploiting Style Latent Flows for Generalizing Deepfake Video Detection](https://mlsanity.com/paper/2403.06592/) (2024)
- [CMUA-Watermark: A Cross-Model Universal Adversarial Watermark for Combating Deepfakes](https://mlsanity.com/paper/2105.10872/) (2021)
- [Anti-Forgery: Towards a Stealthy and Robust DeepFake Disruption Attack via Adversarial Perceptual-aware Perturbations](https://mlsanity.com/paper/2206.00477/) (2022)
- [Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editing](https://mlsanity.com/paper/2411.16832/) (2024)
- [FaceShield: Defending Facial Image against Deepfake Threats](https://mlsanity.com/paper/2412.09921/) (2024)
- [Weakly-supervised Audio Temporal Forgery Localization via Progressive Audio-language Co-learning Network](https://mlsanity.com/paper/2505.01880/) (2025)
- [Diffusion-based Adversarial Identity Manipulation for Facial Privacy Protection](https://mlsanity.com/paper/2504.21646/) (2025)

## Built on by (most cited first, 1 of 1)

- [Adversarial Attacks for Good: A Survey of Proactive Protection across the Visual Content Lifecycle](https://mlsanity.com/paper/2608.04314/) (2026)

## Abstract (arXiv)

The rapid evolution of diffusion models has democratized face swapping but also raises concerns about privacy and identity security. Existing proactive defenses, often adapted from image editing attacks, prove ineffective in this context. We attribute this failure to an oversight of the structural resilience and the unique static conditional guidance mechanism inherent in face swapping systems. To address this, we propose VoidFace, a systemic defense method that views face swapping as a coupled identity pathway. By injecting perturbations at critical bottlenecks, VoidFace induces cascading disruption throughout the pipeline. Specifically, we first introduce localization disruption and identity erasure to degrade physical regression and semantic embeddings, thereby impairing the accurate modeling of the source face. We then intervene in the generative domain by decoupling attention mechanisms to sever identity injection, and corrupting intermediate diffusion features to prevent the reconstruction of source identity. To ensure visual imperceptibility, we perform adversarial search in the latent manifold, guided by a perceptual adaptive strategy to balance attack potency with image quality. Extensive experiments show that VoidFace outperforms existing defenses across various diffusion-based swapping models, while producing adversarial faces with superior visual quality.