mlsanityData to 7 Oct 2026

Stateful Defenses for Machine Learning Models Are Not Yet Secure Against Black-box Attacks

arXiv 2303.06280 ↗First posted 11 Mar 2023cs.CR cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Recent work has proposed stateful defense models (SDMs) as a compelling strategy to defend against a black-box attacker who only has query access to the model, as is common for online machine learning platforms. Such stateful defenses aim to defend against black-box attacks by tracking the query history and detecting and rejecting queries that are "similar" and thus preventing black-box attacks from finding useful gradients and making progress towards finding adversarial attacks within a reasonable query budget. Recent SDMs (e.g., Blacklight and PIHA) have shown remarkable success in defending against state-of-the-art black-box attacks. In this paper, we show that SDMs are highly vulnerable to a new class of adaptive black-box attacks. We propose a novel adaptive black-box attack strategy called Oracle-guided Adaptive Rejection Sampling (OARS) that involves two stages: (1) use initial query patterns to infer key properties about an SDM's defense; and, (2) leverage those extracted properties to design subsequent query patterns to evade the SDM's defense while making progress towards finding adversarial inputs. OARS is broadly applicable as an enhancement to existing black-box attacks - we show how to apply the strategy to enhance six common black-box attacks to be more effective against current class of SDMs. For example, OARS-enhanced versions of black-box attacks improved attack success rate against recent stateful defenses from almost 0% to to almost 100% for multiple datasets within reasonable query budgets.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 26 of its 50 references that are in the map. First posted on arXiv 11 Mar 2023.

Builds on

26 of 50 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  6. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  7. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  8. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  9. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  10. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  11. 2017Adversarial Patchcited by 514
  12. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
14 more
  1. 2017Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Modelscited by 459
  2. 2019HopSkipJumpAttack: A Query-Efficient Decision-Based Attackcited by 328
  3. 2018PRADA: Protecting against DNN Model Stealing Attackscited by 159
  4. 2020QEBA: Query-Efficient Boundary-Based Blackbox Attackcited by 103
  5. 2016Deep Learning for Identifying Metastatic Breast Cancercited by 93
  6. 2020SurFree: a fast surrogate-free black-box attackcited by 59
  7. 2019Stateful Detection of Black-Box Adversarial Attackscited by 55
  8. 2019Parsimonious Black-Box Adversarial Attacks via Efficient Combinatorial Optimizationcited by 49
  9. 2020Blacklight: Scalable Defense for Neural Networks against Query-Based Black-Box Attackscited by 34
  10. 2021Deepfake Videos in the Wild: Analysis and Detectioncited by 23
  11. 2022Adversarial Attack on Attackers: Post-Process to Mitigate Black-Box Score-Based Query Attackscited by 18
  12. 2020AdvMind: Inferring Adversary Intent of Black-Box Attackscited by 12
  13. 2020GRAPHITE: Generating Automatic Physical Examples for Machine-Learning Attacks on Computer Vision Systemscited by 7
  14. 2019PDQ & TMK + PDQF -- A Test Drive of Facebook's Perceptual Hashing Algorithmscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 2; 2026: 3.

10 papers.

  1. 2024L-AutoDA: Leveraging Large Language Models for Automated Decision-based Adversarial Attackscited by 8
  2. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  3. 2023PubDef: Defending Against Transfer Attacks From Public Modelscited by 7
  4. 2024AdvQDet: Detecting Query-Based Adversarial Attacks with Adversarial Contrastive Prompt Tuningcited by 7
  5. 2023The Adaptive Arms Race: Redefining Robustness in AI Securitycited by 4
  6. 2026SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecuritycited by 4
  7. 2024Taking off the Rose-Tinted Glasses: A Critical Look at Adversarial ML Through the Lens of Evasion Attackscited by 1
  8. 2025How stealthy is stealthy? Studying the Efficacy of Black-Box Adversarial Attacks in the Real Worldcited by 0
  9. 2026SoK: Colluding Adversaries in Machine Learning Pipelinescited by 0
  10. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0