mlsanityData to 14 Oct 2026

My Face Is Mine, Not Yours: Facial Protection Against Diffusion Model Face Swapping

arXiv 2505.15336 ↗First posted 21 May 2025cs.CVcited by 5

On the map: Diffusion models → Robustness, privacy and adversarial aspects

Abstract (arXiv)

The proliferation of diffusion-based deepfake technologies poses significant risks for unauthorized and unethical facial image manipulation. While traditional countermeasures have primarily focused on passive detection methods, this paper introduces a novel proactive defense strategy through adversarial attacks that preemptively protect facial images from being exploited by diffusion-based deepfake systems. Existing adversarial protection methods predominantly target conventional generative architectures (GANs, AEs, VAEs) and fail to address the unique challenges presented by diffusion models, which have become the predominant framework for high-quality facial deepfakes. Current diffusion-specific adversarial approaches are limited by their reliance on specific model architectures and weights, rendering them ineffective against the diverse landscape of diffusion-based deepfake implementations. Additionally, they typically employ global perturbation strategies that inadequately address the region-specific nature of facial manipulation in deepfakes.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 43 of its 53 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

43 of 53 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024A Gray-box Attack against Latent Diffusion Model-based Image Editing by Posterior Collapse20 Aug
  2. 2024Attacks and Defenses for Generative Diffusion Models: A Comprehensive Survey6 Aug
  3. 2023DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification27 Oct
  4. 2023Toward effective protection against diffusion based mimicry through score distillation2 Oct
  5. 2023AdvDiff: Generating Unrestricted Adversarial Examples using Diffusion Models24 Jul
  6. 2023Mist: Towards Improved Adversarial Examples for Diffusion Models22 May
  7. 2023Diffusion Models for Imperceptible and Transferable Adversarial Attack14 May
  8. 2023TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets10 Mar
  9. 2023Raising the Cost of Malicious AI-Powered Image Editing13 Feb
  10. 2023Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples9 Feb
  11. 2021Adversarial purification with Score-based generative models11 Jun

Other references · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  12. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
20 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  2. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  3. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  4. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  5. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  6. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
  7. 2019FSGAN: Subject Agnostic Face Swapping and Reenactmentcited by 320
  8. 2021SimSwap: An Efficient Framework For High Fidelity Face Swappingcited by 243
  9. 2022TruFor: Leveraging all-round clues for trustworthy image forgery detection and localizationcited by 130
  10. 2021One Shot Face Swapping on Megapixelscited by 105
  11. 2021Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forwardcited by 82
  12. 2022DiffFace: Diffusion-based Face Swapping with Facial Guidancecited by 77
  13. 2024Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Controlcited by 67
  14. 2023Diffusion-Based Adversarial Sample Generation for Improved Stealthiness and Controllabilitycited by 66
  15. 2022StyleSwap: Style-Based Generator Empowers Robust Face Swappingcited by 39
  16. 2024Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Modelscited by 36
  17. 2022Restricted Black-box Adversarial Attack Against DeepFake Face Swappingcited by 25
  18. 2023A White-Box False Positive Adversarial Attack Method on Contrastive Loss Based Offline Handwritten Signature Verification Modelscited by 15
  19. 2024Instant Adversarial Purification with Adversarial Consistency Distillationcited by 14
  20. 2024Artwork Protection Against Neural Style Transfer Using Locally Adaptive Adversarial Color Attackcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Beyond Vulnerabilities: A Survey of Adversarial Attacks as Both Threats and Defenses in Computer Vision Systemscited by 4
  2. 2026Adversarial Attacks for Good: A Survey of Proactive Protection across the Visual Content Lifecyclecited by 2
  3. 2026Beauty and the Beast: Imperceptible Perturbations Against Diffusion-Based Face Swapping via Directional Attribute Editingcited by 1
  4. 2026A Survey on Adversarial Attacks and Defenses for Diffusion Models Across Multiple Modalitiescited by 0
  5. 2026Protective Perturbations Must Survive the Resize: Scale-Robust Image Immunization against Malicious Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024FaceShield: Defending Facial Image against Deepfake Threats
  2. 2026Diffusion-Guided Adversarial Perturbation Injection for Generalizable Defense Against Facial Manipulations
  3. 2024DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion
  4. 2023DiffProtect: Generate Adversarial Examples with Diffusion Models for Facial Privacy Protection
  5. 2025Diffusion-based Adversarial Identity Manipulation for Facial Privacy Protection
  6. 2023On the Vulnerability of DeepFake Detectors to Attacks Generated by Denoising Diffusion Models
  7. 2025Enhancing Facial Privacy Protection via Weakening Diffusion Purification
  8. 2024LDFaceNet: Latent Diffusion-based Network for High-Fidelity Deepfake Generation