# Learning Black-Box Attackers with Transferable Priors and Query Feedback

arXiv: https://arxiv.org/abs/2010.11742 · HTML: https://mlsanity.com/paper/2010.11742/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2020.

## Built on by (most cited first, 30 of 32)

- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [Shadows can be Dangerous: Stealthy and Effective Physical-world Adversarial Attack by Natural Phenomenon](https://mlsanity.com/paper/2203.03818/) (2022)
- [Shape-invariant 3D Adversarial Point Clouds](https://mlsanity.com/paper/2203.04041/) (2022)
- [Towards Understanding and Boosting Adversarial Transferability from a Distribution Perspective](https://mlsanity.com/paper/2210.04213/) (2022)
- [Black-Box Dissector: Towards Erasing-based Hard-Label Model Stealing Attack](https://mlsanity.com/paper/2105.00623/) (2021)
- [QAIR: Practical Query-efficient Black-Box Attacks for Image Retrieval](https://mlsanity.com/paper/2103.02927/) (2021)
- [Jigsaw-ViT: Learning Jigsaw Puzzles in Vision Transformer](https://mlsanity.com/paper/2207.11971/) (2022)
- [Adversarially-Aware Robust Object Detector](https://mlsanity.com/paper/2207.06202/) (2022)
- [Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective](https://mlsanity.com/paper/2302.09457/) (2023)
- [A Survey of Robustness and Safety of 2D and 3D Deep Learning Models Against Adversarial Attacks](https://mlsanity.com/paper/2310.00633/) (2023)
- [Attacking deep networks with surrogate-based adversarial black-box methods is easy](https://mlsanity.com/paper/2203.08725/) (2022)
- [Blackbox Attacks via Surrogate Ensemble Search](https://mlsanity.com/paper/2208.03610/) (2022)
- [Efficient Black-box Adversarial Attacks via Bayesian Optimization Guided by a Function Prior](https://mlsanity.com/paper/2405.19098/) (2024)
- [Hard-label based Small Query Black-box Adversarial Attack](https://mlsanity.com/paper/2403.06014/) (2024)
- [JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models](https://mlsanity.com/paper/2505.19610/) (2025)
- [DST: Dynamic Substitute Training for Data-free Black-box Attack](https://mlsanity.com/paper/2204.00972/) (2022)
- [Decompiling x86 Deep Neural Network Executables](https://mlsanity.com/paper/2210.01075/) (2022)
- [Transferable Physical Attack against Object Detection with Separable Attention](https://mlsanity.com/paper/2205.09592/) (2022)
- [Adversarial for Good? How the Adversarial ML Community's Values Impede Socially Beneficial Uses of Attacks](https://mlsanity.com/paper/2107.10302/) (2021)
- [Query Attack by Multi-Identity Surrogates](https://mlsanity.com/paper/2105.15010/) (2021)
- [Reinforcement Learning Platform for Adversarial Black-box Attacks with Custom Distortion Filters](https://mlsanity.com/paper/2501.14122/) (2025)
- [Enhancing the Transferability via Feature-Momentum Adversarial Attack](https://mlsanity.com/paper/2204.10606/) (2022)
- [Query-Efficient Hard-Label Black-Box Attack against Vision Transformers](https://mlsanity.com/paper/2407.00389/) (2024)
- [GSBAK: top-K Geometric Score-based Black-box Attack](https://mlsanity.com/paper/2503.12827/) (2025)
- [Tracklet-Switch Adversarial Attack against Pedestrian Multi-Object Tracking Trackers](https://mlsanity.com/paper/2111.08954/) (2021)
- [Eidos: Efficient, Imperceptible Adversarial 3D Point Clouds](https://mlsanity.com/paper/2405.14210/) (2024)
- [Gradient Aligned Attacks via a Few Queries](https://mlsanity.com/paper/2205.09518/) (2022)
- [CARNet: Collaborative Adversarial Resilience for Robust Underwater Image Enhancement and Perception](https://mlsanity.com/paper/2309.01102/) (2023)
- [Counter-Samples: A Stateless Strategy to Neutralize Black Box Adversarial Attacks](https://mlsanity.com/paper/2403.10562/) (2024)
- [DataCook: Crafting Anti-Adversarial Examples for Healthcare Data Copyright Protection](https://mlsanity.com/paper/2403.17755/) (2024)

## Abstract (arXiv)

This paper addresses the challenging black-box adversarial attack problem, where only classification confidence of a victim model is available. Inspired by consistency of visual saliency between different vision models, a surrogate model is expected to improve the attack performance via transferability. By combining transferability-based and query-based black-box attack, we propose a surprisingly simple baseline approach (named SimBA++) using the surrogate model, which significantly outperforms several state-of-the-art methods. Moreover, to efficiently utilize the query feedback, we update the surrogate model in a novel learning scheme, named High-Order Gradient Approximation (HOGA). By constructing a high-order gradient computation graph, we update the surrogate model to approximate the victim model in both forward and backward pass. The SimBA++ and HOGA result in Learnable Black-Box Attack (LeBA), which surpasses previous state of the art by considerable margins: the proposed LeBA significantly reduces queries, while keeping higher attack success rates close to 100% in extensive ImageNet experiments, including attacking vision benchmarks and defensive models. Code is open source at https://github.com/TrustworthyDL/LeBA.