# ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models

arXiv: https://arxiv.org/abs/1708.03999 · HTML: https://mlsanity.com/paper/1708.03999/

## Key facts

- Cited by 666 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2017.

## Built on by (most cited first, 30 of 666)

- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [On Evaluating Adversarial Robustness of Large Vision-Language Models](https://mlsanity.com/paper/2305.16934/) (2023)
- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [Open Sesame! Universal Black Box Jailbreaking of Large Language Models](https://mlsanity.com/paper/2309.01446/) (2023)
- [Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark](https://mlsanity.com/paper/2402.11592/) (2024)
- [Shadows can be Dangerous: Stealthy and Effective Physical-world Adversarial Attack by Natural Phenomenon](https://mlsanity.com/paper/2203.03818/) (2022)
- [Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning](https://mlsanity.com/paper/2205.01992/) (2022)
- [Diffusion Models for Imperceptible and Transferable Adversarial Attack](https://mlsanity.com/paper/2305.08192/) (2023)
- [Adversarial Sticker: A Stealthy Attack Method in the Physical World](https://mlsanity.com/paper/2104.06728/) (2021)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [Rethinking Model Ensemble in Transfer-based Adversarial Attacks](https://mlsanity.com/paper/2303.09105/) (2023)
- [Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protection](https://mlsanity.com/paper/2210.00875/) (2022)
- [Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detection](https://mlsanity.com/paper/2201.08970/) (2022)
- [A Pilot Study of Query-Free Adversarial Attack against Stable Diffusion](https://mlsanity.com/paper/2303.16378/) (2023)
- [DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training](https://mlsanity.com/paper/2310.02025/) (2023)
- [Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes](https://mlsanity.com/paper/2403.00867/) (2024)
- [Bad Characters: Imperceptible NLP Attacks](https://mlsanity.com/paper/2106.09898/) (2021)
- [On Improving Adversarial Transferability of Vision Transformers](https://mlsanity.com/paper/2106.04169/) (2021)
- [InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models](https://mlsanity.com/paper/2306.03082/) (2023)
- [HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noise](https://mlsanity.com/paper/2110.11417/) (2021)
- [Black Box Adversarial Prompting for Foundation Models](https://mlsanity.com/paper/2302.04237/) (2023)
- [Query-Based Adversarial Prompt Generation](https://mlsanity.com/paper/2402.12329/) (2024)
- [Relating Adversarially Robust Generalization to Flat Minima](https://mlsanity.com/paper/2104.04448/) (2021)
- [Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs](https://mlsanity.com/paper/2110.02797/) (2021)
- [Meta Gradient Adversarial Attack](https://mlsanity.com/paper/2108.04204/) (2021)
- [Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning](https://mlsanity.com/paper/2402.15751/) (2024)
- [Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer](https://mlsanity.com/paper/2402.15173/) (2024)
- [Adversarial Example Detection for DNN Models: A Review and Experimental Comparison](https://mlsanity.com/paper/2105.00203/) (2021)
- [Shape-invariant 3D Adversarial Point Clouds](https://mlsanity.com/paper/2203.04041/) (2022)

## Abstract (arXiv)

Deep neural networks (DNNs) are one of the most prominent technologies of our time, as they achieve state-of-the-art performance in many machine learning tasks, including but not limited to image classification, text mining, and speech processing. However, recent research on DNNs has indicated ever-increasing concern on the robustness to adversarial examples, especially for security-critical tasks such as traffic sign identification for autonomous driving. Studies have unveiled the vulnerability of a well-trained DNN by demonstrating the ability of generating barely noticeable (to both human and machines) adversarial images that lead to misclassification. Furthermore, researchers have shown that these adversarial images are highly transferable by simply training and attacking a substitute model built upon the target model, known as a black-box attack to DNNs. Similar to the setting of training substitute models, in this paper we propose an effective black-box attack that also only has access to the input (images) and the output (confidence scores) of a targeted DNN. However, different from leveraging attack transferability from substitute models, we propose zeroth order optimization (ZOO) based attacks to directly estimate the gradients of the targeted DNN for generating adversarial examples. We use zeroth order stochastic coordinate descent along with dimension reduction, hierarchical attack and importance sampling techniques to efficiently attack black-box models. By exploiting zeroth order optimization, improved attacks to the targeted DNN can be accomplished, sparing the need for training substitute models and avoiding the loss in attack transferability. Experimental results on MNIST, CIFAR10 and ImageNet show that the proposed ZOO attack is as effective as the state-of-the-art white-box attack and significantly outperforms existing black-box attacks via substitute models.