# Attacking deep networks with surrogate-based adversarial black-box methods is easy

arXiv: https://arxiv.org/abs/2203.08725 · HTML: https://mlsanity.com/paper/2203.08725/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 46 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 30 of 37)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [Delving into Transferable Adversarial Examples and Black-box Attacks](https://mlsanity.com/paper/1611.02770/) (2016)
- [Black-box Adversarial Attacks with Limited Queries and Information](https://mlsanity.com/paper/1804.08598/) (2018)
- [Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models](https://mlsanity.com/paper/1712.04248/) (2017)
- [Simple Black-box Adversarial Attacks](https://mlsanity.com/paper/1905.07121/) (2019)
- [Spatially Transformed Adversarial Examples](https://mlsanity.com/paper/1801.02612/) (2018)
- [Generating Natural Adversarial Examples](https://mlsanity.com/paper/1710.11342/) (2017)
- [Prior Convictions: Black-Box Adversarial Attacks with Bandits and Priors](https://mlsanity.com/paper/1807.07978/) (2018)
- [AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networks](https://mlsanity.com/paper/1805.11770/) (2018)
- [Improving Black-box Adversarial Attacks with a Transfer-based Prior](https://mlsanity.com/paper/1906.06919/) (2019)
- [NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks](https://mlsanity.com/paper/1905.00441/) (2019)
- [Semantic Adversarial Examples](https://mlsanity.com/paper/1804.00499/) (2018)
- [Low Frequency Adversarial Perturbation](https://mlsanity.com/paper/1809.08758/) (2018)
- [Black-Box Adversarial Attack with Transferable Model-based Embedding](https://mlsanity.com/paper/1911.07140/) (2019)
- [Parsimonious Black-Box Adversarial Attacks via Efficient Combinatorial Optimization](https://mlsanity.com/paper/1905.06635/) (2019)
- [Guessing Smart: Biased Sampling for Efficient Black-Box Adversarial Attacks](https://mlsanity.com/paper/1812.09803/) (2018)
- [Unrestricted Adversarial Examples](https://mlsanity.com/paper/1809.08352/) (2018)
- [Curls & Whey: Boosting Black-Box Adversarial Attacks](https://mlsanity.com/paper/1904.01160/) (2019)
- [AdvFlow: Inconspicuous Black-box Adversarial Attacks using Normalizing Flows](https://mlsanity.com/paper/2007.07435/) (2020)

## Built on by (most cited first, 18 of 18)

- [PAL: Proxy-Guided Black-Box Attack on Large Language Models](https://mlsanity.com/paper/2402.09674/) (2024)
- [Ensemble-based Blackbox Attacks on Dense Prediction](https://mlsanity.com/paper/2303.14304/) (2023)
- [Blackbox Attacks via Surrogate Ensemble Search](https://mlsanity.com/paper/2208.03610/) (2022)
- [Boosting Black-box Attack to Deep Neural Networks with Conditional Diffusion Models](https://mlsanity.com/paper/2310.07492/) (2023)
- [Efficient Black-box Adversarial Attacks via Bayesian Optimization Guided by a Function Prior](https://mlsanity.com/paper/2405.19098/) (2024)
- [SoK: Pitfalls in Evaluating Black-Box Attacks](https://mlsanity.com/paper/2310.17534/) (2023)
- [DOEPatch: Dynamically Optimized Ensemble Model for Adversarial Patches Generation](https://mlsanity.com/paper/2312.16907/) (2023)
- [GreedyPixel: Fine-Grained Black-Box Adversarial Attack Via Greedy Algorithm](https://mlsanity.com/paper/2501.14230/) (2025)
- [Adversarial Attacks on Hidden Tasks in Multi-Task Learning](https://mlsanity.com/paper/2405.15244/) (2024)
- [DifAttack++: Query-Efficient Black-Box Adversarial Attack via Hierarchical Disentangled Feature Space in Cross-Domain](https://mlsanity.com/paper/2406.03017/) (2024)
- [Common Knowledge Learning for Generating Transferable Adversarial Examples](https://mlsanity.com/paper/2307.00274/) (2023)
- [DifAttack: Query-Efficient Black-Box Attack via Disentangled Feature Space](https://mlsanity.com/paper/2309.14585/) (2023)
- [Gradient Aligned Attacks via a Few Queries](https://mlsanity.com/paper/2205.09518/) (2022)
- [Query Efficient Cross-Dataset Transferable Black-Box Attack on Action Recognition](https://mlsanity.com/paper/2211.13171/) (2022)
- [PEAS: A Strategy for Crafting Transferable Adversarial Examples](https://mlsanity.com/paper/2410.15409/) (2024)
- [With Great Backbones Comes Great Adversarial Transferability](https://mlsanity.com/paper/2501.12275/) (2025)
- [Exploiting Edge Features for Transferable Adversarial Attacks in Distributed Machine Learning](https://mlsanity.com/paper/2507.07259/) (2025)
- [Learning with Bilevel-Minimax Optimization for Efficient and Reliable Transfer Attacks](https://mlsanity.com/paper/2608.11815/) (2026)

## Abstract (arXiv)

A recent line of work on black-box adversarial attacks has revived the use of transfer from surrogate models by integrating it into query-based search. However, we find that existing approaches of this type underperform their potential, and can be overly complicated besides. Here, we provide a short and simple algorithm which achieves state-of-the-art results through a search which uses the surrogate network's class-score gradients, with no need for other priors or heuristics. The guiding assumption of the algorithm is that the studied networks are in a fundamental sense learning similar functions, and that a transfer attack from one to the other should thus be fairly "easy". This assumption is validated by the extremely low query counts and failure rates achieved: e.g. an untargeted attack on a VGG-16 ImageNet network using a ResNet-152 as the surrogate yields a median query count of 6 at a success rate of 99.9%. Code is available at https://github.com/fiveai/GFCS.