# Adversarial Attacks and Identity Leakage in De-Identification Systems: An Empirical Study

arXiv: https://arxiv.org/abs/2609.27022 · HTML: https://mlsanity.com/paper/2609.27022/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 56 references that are in the map.
- First posted on arXiv 22 Sep 2026.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)
- [Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/1801.00553/) (2018)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [SimSwap: An Efficient Framework For High Fidelity Face Swapping](https://mlsanity.com/paper/2106.06340/) (2021)
- [Adversarial Attacks and Defenses in Images, Graphs and Text: A Review](https://mlsanity.com/paper/1909.08072/) (2019)
- [AdvHat: Real-world adversarial attack on ArcFace Face ID system](https://mlsanity.com/paper/1908.08705/) (2019)
- [ElasticFace: Elastic Margin Loss for Deep Face Recognition](https://mlsanity.com/paper/2109.09416/) (2021)
- [On the Robustness of Semantic Segmentation Models to Adversarial Attacks](https://mlsanity.com/paper/1711.09856/) (2017)
- [DeepPrivacy: A Generative Adversarial Network for Face Anonymization](https://mlsanity.com/paper/1909.04538/) (2019)
- [Adv-Makeup: A New Imperceptible and Transferable Attack on Face Recognition](https://mlsanity.com/paper/2105.03162/) (2021)
- [Rethinking Model Ensemble in Transfer-based Adversarial Attacks](https://mlsanity.com/paper/2303.09105/) (2023)
- [Towards Transferable Adversarial Attack against Deep Face Recognition](https://mlsanity.com/paper/2004.05790/) (2020)
- [Disrupting Deepfakes: Adversarial Attacks Against Conditional Image Translation Networks and Facial Manipulation Systems](https://mlsanity.com/paper/2003.01279/) (2020)
- [DeepPrivacy2: Towards Realistic Full-Body Anonymization](https://mlsanity.com/paper/2211.09454/) (2022)
- [AnonymousNet: Natural Face De-Identification with Measurable Privacy](https://mlsanity.com/paper/1904.12620/) (2019)
- [Live Face De-Identification in Video](https://mlsanity.com/paper/1911.08348/) (2019)
- [FaceDancer: Pose- and Occlusion-Aware High Fidelity Face Swapping](https://mlsanity.com/paper/2210.10473/) (2022)
- [A Frank-Wolfe Framework for Efficient and Effective Adversarial Attacks](https://mlsanity.com/paper/1811.10828/) (2018)
- [Robust Physical-World Attacks on Face Recognition](https://mlsanity.com/paper/2109.09320/) (2021)
- [Restricted Black-box Adversarial Attack Against DeepFake Face Swapping](https://mlsanity.com/paper/2204.12347/) (2022)

## Abstract (arXiv)

In this paper, we investigate the impact of adversarial attacks on identity encoders within a realistic de-identification framework. Our experiments show that the transferability of attacks transfers from an external surrogate model to the system model (e.g., CosFace to ArcFace) allows the adversary to cause identity information to leak in a sufficiently sensitive face recognition system. We present experimental evidence and propose strategies to mitigate this vulnerability. Specifically, we show how fine-tuning on adversarial examples helps to mitigate this effect for distortion-based attacks (i.e., snow, fog, etc.), while a simple low-pass filter can attenuate the effect of adversarial noise without affecting the de-identified images. Our mitigation results in a de-identification system that preserves its functionality while being significantly more robust to adversarial noise.