mlsanityData to 7 Oct 2026

Performance Evaluation of Adversarial Attacks: Discrepancies and Solutions

arXiv 2104.11103 ↗First posted 22 Apr 2021cs.LG cs.CRcited by 7

Not on the map. Why?

Abstract (arXiv)

Recently, adversarial attack methods have been developed to challenge the robustness of machine learning models. However, mainstream evaluation criteria experience limitations, even yielding discrepancies among results under different settings. By examining various attack algorithms, including gradient-based and query-based attacks, we notice the lack of a consensus on a uniform standard for unbiased performance evaluation. Accordingly, we propose a Piece-wise Sampling Curving (PSC) toolkit to effectively address the aforementioned discrepancy, by generating a comprehensive comparison among adversaries in a given range. In addition, the PSC toolkit offers options for balancing the computational cost and evaluation effectiveness. Experimental results demonstrate our PSC toolkit presents comprehensive comparisons of attack algorithms, significantly reducing discrepancies in practice.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 36 of its 40 references that are in the map. First posted on arXiv 22 Apr 2021.

Builds on

36 of 40 references.

References · 36

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  5. 2016Adversarial examples in the physical worldcited by 1,762
  6. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  7. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  8. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  9. 2016Universal adversarial perturbationscited by 774
  10. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  11. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  12. 2016Stealing Machine Learning Models via Prediction APIscited by 602
24 more
  1. 2017One pixel attack for fooling deep neural networkscited by 579
  2. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  3. 2017Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Modelscited by 459
  4. 2019Simple Black-box Adversarial Attackscited by 249
  5. 2018Adversarial Robustness Toolbox v1.0.0cited by 214
  6. 2017Generative Adversarial Perturbationscited by 153
  7. 2018Prior Convictions: Black-Box Adversarial Attacks with Bandits and Priorscited by 145
  8. 2018AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networkscited by 137
  9. 2018Query-Efficient Hard-label Black-box Attack:An Optimization-based Approachcited by 130
  10. 2019Sign-OPT: A Query-Efficient Hard-label Adversarial Attackcited by 118
  11. 2018Decoupling Direction and Norm for Efficient Gradient-Based L2 Adversarial Attacks and Defensescited by 109
  12. 2019NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networkscited by 94
  13. 2020RayS: A Ray Searching Method for Hard-label Adversarial Attackcited by 94
  14. 2018Generalizable Data-free Objective for Crafting Universal Adversarial Perturbationscited by 88
  15. 2019advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorchcited by 69
  16. 2017Fast Feature Fool: A data independent approach to universal adversarial perturbationscited by 65
  17. 2020DaST: Data-free Substitute Training for Adversarial Attackscited by 60
  18. 2015Adversarial Manipulation of Deep Representationscited by 56
  19. 2017Art of singular vectors and universal adversarial perturbationscited by 49
  20. 2019Accurate, reliable and fast robustness evaluationcited by 43
  21. 2016Adversarial Diversity and Hard Positive Generationcited by 35
  22. 2018Ask, Acquire, and Attack: Data-free UAP Generation using Class Impressionscited by 31
  23. 2019Simple iterative method for generating targeted universal adversarial perturbationscited by 13
  24. 2020Decision-based Universal Adversarial Attackcited by 6

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 3.

7 papers.

  1. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  2. 2022A Survey of Adversarial Defences and Robustness in NLPcited by 22
  3. 2023Robust Text Classification: Analyzing Prototype-Based Networkscited by 3
  4. 2025The Structural Safety Generalization Problemcited by 2
  5. 2024Post-Hoc Robustness Enhancement in Graph Neural Networks with Conditional Random Fieldscited by 0
  6. 2025Towards Assessing Deep Learning Test Input Generatorscited by 0
  7. 2025The Butterfly Effect in Pathology: Exploring Security in Pathology Foundation Modelscited by 0