# Data Poisoning Won't Save You From Facial Recognition

arXiv: https://arxiv.org/abs/2106.14851 · HTML: https://mlsanity.com/paper/2106.14851/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 47 of its 80 references that are in the map.
- First posted on arXiv 28 Jun 2021.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [Delving into Transferable Adversarial Examples and Black-box Attacks](https://mlsanity.com/paper/1611.02770/) (2016)
- [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014)
- [Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples](https://mlsanity.com/paper/1605.07277/) (2016)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [Fooling automated surveillance cameras: adversarial patches to attack person detection](https://mlsanity.com/paper/1904.08653/) (2019)

## Built on by (most cited first, 30 of 45)

- [Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models](https://mlsanity.com/paper/2302.04222/) (2023)
- [Raising the Cost of Malicious AI-Powered Image Editing](https://mlsanity.com/paper/2302.06588/) (2023)
- [An Adversarial Perspective on Machine Unlearning for AI Safety](https://mlsanity.com/paper/2409.18025/) (2024)
- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [On Evaluating the Durability of Safeguards for Open-Weight LLMs](https://mlsanity.com/paper/2412.07097/) (2024)
- [Image Shortcut Squeezing: Countering Perturbative Availability Poisons with Compression](https://mlsanity.com/paper/2301.13838/) (2023)
- [Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion?](https://mlsanity.com/paper/2312.00084/) (2023)
- [IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AI](https://mlsanity.com/paper/2310.19248/) (2023)
- [Indiscriminate Data Poisoning Attacks on Neural Networks](https://mlsanity.com/paper/2204.09092/) (2022)
- [What Can We Learn from Unlearnable Datasets?](https://mlsanity.com/paper/2305.19254/) (2023)
- [The Devil's Advocate: Shattering the Illusion of Unexploitable Data using Diffusion Models](https://mlsanity.com/paper/2303.08500/) (2023)
- [SoK: Anti-Facial Recognition Technology](https://mlsanity.com/paper/2112.04558/) (2021)
- [FACE-AUDITOR: Data Auditing in Facial Recognition Systems](https://mlsanity.com/paper/2304.02782/) (2023)
- [JPEG Compressed Images Can Bypass Protections Against AI Editing](https://mlsanity.com/paper/2304.02234/) (2023)
- [Semantic Deep Hiding for Robust Unlearnable Examples](https://mlsanity.com/paper/2406.17349/) (2024)
- [Conflicting Interactions Among Protection Mechanisms for Machine Learning Models](https://mlsanity.com/paper/2207.01991/) (2022)
- [Data Isotopes for Data Provenance in DNNs](https://mlsanity.com/paper/2208.13893/) (2022)
- [Dormant: Defending against Pose-driven Human Image Animation](https://mlsanity.com/paper/2409.14424/) (2024)
- [Fairness Properties of Face Recognition and Obfuscation Systems](https://mlsanity.com/paper/2108.02707/) (2021)
- [Face Recognition in the age of CLIP & Billion image datasets](https://mlsanity.com/paper/2301.07315/) (2023)
- [Adversarial Machine Learning for Social Good: Reframing the Adversary as an Ally](https://mlsanity.com/paper/2310.03614/) (2023)
- [SoK: Pitfalls in Evaluating Black-Box Attacks](https://mlsanity.com/paper/2310.17534/) (2023)
- [Fantomas: Understanding Face Anonymization Reversibility](https://mlsanity.com/paper/2210.10651/) (2022)
- [ConfounderGAN: Protecting Image Data Privacy with Causal Confounder](https://mlsanity.com/paper/2212.01767/) (2022)
- [You Don't Need Robust Machine Learning to Manage Adversarial Attack Risks](https://mlsanity.com/paper/2306.09951/) (2023)
- [BridgePure: Limited Protection Leakage Can Break Black-Box Data Protection](https://mlsanity.com/paper/2412.21061/) (2024)
- [CamPro: Camera-based Anti-Facial Recognition](https://mlsanity.com/paper/2401.00151/) (2023)
- [Crafter: Facial Feature Crafting against Inversion-based Identity Theft on Deep Models](https://mlsanity.com/paper/2401.07205/) (2024)
- [DivTrackee versus DynTracker: Promoting Diversity in Anti-Facial Recognition against Dynamic FR Strategy](https://mlsanity.com/paper/2501.06533/) (2025)
- [Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2502.00451/) (2025)

## Abstract (arXiv)

Data poisoning has been proposed as a compelling defense against facial recognition models trained on Web-scraped pictures. Users can perturb images they post online, so that models will misclassify future (unperturbed) pictures. We demonstrate that this strategy provides a false sense of security, as it ignores an inherent asymmetry between the parties: users' pictures are perturbed once and for all before being published (at which point they are scraped) and must thereafter fool all future models -- including models trained adaptively against the users' past attacks, or models that use technologies discovered after the attack. We evaluate two systems for poisoning attacks against large-scale facial recognition, Fawkes (500'000+ downloads) and LowKey. We demonstrate how an "oblivious" model trainer can simply wait for future developments in computer vision to nullify the protection of pictures collected in the past. We further show that an adversary with black-box access to the attack can (i) train a robust model that resists the perturbations of collected pictures and (ii) detect poisoned pictures uploaded online. We caution that facial recognition poisoning will not admit an "arms race" between attackers and defenders. Once perturbed pictures are scraped, the attack cannot be changed so any future successful defense irrevocably undermines users' privacy.