# Post-breach Recovery: Protection against White-box Adversarial Examples for Leaked DNN Models

arXiv: https://arxiv.org/abs/2205.10686 · HTML: https://mlsanity.com/paper/2205.10686/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2022.

## Built on by (most cited first, 7 of 7)

- [Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models](https://mlsanity.com/paper/2302.04222/) (2023)
- ["Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice](https://mlsanity.com/paper/2212.14315/) (2022)
- [Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2305.06360/) (2023)
- [Self-Deception: Reverse Penetrating the Semantic Firewall of Large Language Models](https://mlsanity.com/paper/2308.11521/) (2023)
- [Unlearnable Examples For Time Series](https://mlsanity.com/paper/2402.02028/) (2024)
- [Towards Scalable and Robust Model Versioning](https://mlsanity.com/paper/2401.09574/) (2024)
- [Fight Fire with Fire: Combating Adversarial Patch Attacks using Pattern-randomized Defensive Patches](https://mlsanity.com/paper/2311.06122/) (2023)

## Abstract (arXiv)

Server breaches are an unfortunate reality on today's Internet. In the context of deep neural network (DNN) models, they are particularly harmful, because a leaked model gives an attacker "white-box" access to generate adversarial examples, a threat model that has no practical robust defenses. For practitioners who have invested years and millions into proprietary DNNs, e.g. medical imaging, this seems like an inevitable disaster looming on the horizon. In this paper, we consider the problem of post-breach recovery for DNN models. We propose Neo, a new system that creates new versions of leaked models, alongside an inference time filter that detects and removes adversarial examples generated on previously leaked models. The classification surfaces of different model versions are slightly offset (by introducing hidden distributions), and Neo detects the overfitting of attacks to the leaked model used in its generation. We show that across a variety of tasks and attack methods, Neo is able to filter out attacks from leaked models with very high accuracy, and provides strong protection (7--10 recoveries) against attackers who repeatedly breach the server. Neo performs well against a variety of strong adaptive attacks, dropping slightly in # of breaches recoverable, and demonstrates potential as a complement to DNN defenses in the wild.