# BlackboxBench: A Comprehensive Benchmark of Black-box Adversarial Attacks

arXiv: https://arxiv.org/abs/2312.16979 · HTML: https://mlsanity.com/paper/2312.16979/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2023.

## Built on by (most cited first, 21 of 21)

- [One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIP](https://mlsanity.com/paper/2505.19840/) (2025)
- [Adversarial Training for Defense Against Label Poisoning Attacks](https://mlsanity.com/paper/2502.17121/) (2025)
- [StealthAttack: Robust 3D Gaussian Splatting Poisoning via Density-Guided Illusions](https://mlsanity.com/paper/2510.02314/) (2025)
- [DifAttack++: Query-Efficient Black-Box Adversarial Attack via Hierarchical Disentangled Feature Space in Cross-Domain](https://mlsanity.com/paper/2406.03017/) (2024)
- [Deciphering the Definition of Adversarial Robustness for post-hoc OOD Detectors](https://mlsanity.com/paper/2406.15104/) (2024)
- [Model Mimic Attack: Knowledge Distillation for Provably Transferable Adversarial Examples](https://mlsanity.com/paper/2410.15889/) (2024)
- [RobustBlack: Challenging Black-Box Adversarial Attacks on State-of-the-Art Defenses](https://mlsanity.com/paper/2412.20987/) (2024)
- [GenoArmory: A Unified Evaluation Framework for Adversarial Attacks on Genomic Foundation Models](https://mlsanity.com/paper/2505.10983/) (2025)
- [Consensus-based optimization for closed-box adversarial attacks and a connection to evolution strategies](https://mlsanity.com/paper/2506.24048/) (2025)
- [VidModEx: Interpretable and Efficient Black Box Model Extraction for High-Dimensional Spaces](https://mlsanity.com/paper/2408.02140/) (2024)
- [Adversarial Examples in Environment Perception for Automated Driving (Review)](https://mlsanity.com/paper/2504.08414/) (2025)
- [SEGA: A Transferable Signed Ensemble Gaussian Black-Box Attack against No-Reference Image Quality Assessment Models](https://mlsanity.com/paper/2509.18546/) (2025)
- [Transferable Physical-World Adversarial Patches Against Object Detection in Autonomous Driving](https://mlsanity.com/paper/2604.23105/) (2026)
- [Random Logit Scaling: Defending Deep Neural Networks Against Black-Box Score-Based Adversarial Example Attacks](https://mlsanity.com/paper/2607.14921/) (2026)
- [PADetBench: Towards Benchmarking Physical Attacks against Object Detection](https://mlsanity.com/paper/2408.09181/) (2024)
- [Seeking Flat Minima over Diverse Surrogates for Improved Adversarial Transferability: A Theoretical Framework and Algorithmic Instantiation](https://mlsanity.com/paper/2504.16474/) (2025)
- [On the Adversarial Robustness of Learning-based Conformal Novelty Detection](https://mlsanity.com/paper/2510.00463/) (2025)
- [Dashed Line Defense: Plug-And-Play Defense Against Adaptive Score-Based Query Attacks](https://mlsanity.com/paper/2602.08679/) (2026)
- [LLM-Agnostic Semantic Representation Attack](https://mlsanity.com/paper/2605.08898/) (2026)
- [Adversarial Attacks on Online Handwriting using Salience-based Temporal Editing](https://mlsanity.com/paper/2607.12500/) (2026)
- [Large-scale Testing Global Optimization Methods with Black-box Adversarial Attacks](https://mlsanity.com/paper/2608.13296/) (2026)

## Abstract (arXiv)

Adversarial examples are well-known tools to evaluate the vulnerability of deep neural networks (DNNs). Although lots of adversarial attack algorithms have been developed, it's still challenging in the practical scenario that the model's parameters and architectures are inaccessible to the attacker/evaluator, i.e., black-box adversarial attacks. Due to the practical importance, there has been rapid progress from recent algorithms, reflected by the quick increase in attack success rate and quick decrease in query numbers to the target model. However, there lacks thorough evaluations and comparisons among these algorithms, causing difficulties in tracking the real progress, analyzing advantages and disadvantages of different technical routes, as well as designing future development roadmap of this field. Thus, we aim at building a comprehensive benchmark of black-box adversarial attacks, called BlackboxBench. It mainly provides: 1) a unified, extensible and modular-based codebase, implementing 29 query-based attack algorithms and 30 transfer-based attack algorithms; 2) comprehensive evaluations: we evaluate the implemented algorithms against several mainstreaming model architectures on 2 widely used datasets (CIFAR-10 and a subset of ImageNet), leading to 14,950 evaluations in total; 3) thorough analysis and new insights, as well analytical tools. The website and source codes of BlackboxBench are available at https://blackboxbenchmark.github.io/ and https://github.com/SCLBD/BlackboxBench/, respectively.