mlsanityData to 7 Oct 2026

The Adaptive Arms Race: Redefining Robustness in AI Security

arXiv 2312.13435 ↗First posted 20 Dec 2023cs.AI cs.CRcited by 4

Not on the map. Why?

Abstract (arXiv)

Despite considerable efforts on making them robust, real-world AI-based systems remain vulnerable to decision based attacks, as definitive proofs of their operational robustness have so far proven intractable. Canonical robustness evaluation relies on adaptive attacks, which leverage complete knowledge of the defense and are tailored to bypass it. This work broadens the notion of adaptivity, which we employ to enhance both attacks and defenses, showing how they can benefit from mutual learning through interaction. We introduce a framework for adaptively optimizing black-box attacks and defenses under the competitive game they form. To assess robustness reliably, it is essential to evaluate against realistic and worst-case attacks. We thus enhance attacks and their evasive arsenal together using RL, apply the same principle to defenses, and evaluate them first independently and then jointly under a multi-agent perspective. We find that active defenses, those that dynamically control system responses, are an essential complement to model hardening against decision-based attacks; that these defenses can be circumvented by adaptive attacks, something that elicits defenses being adaptive too. Our findings, supported by an extensive theoretical and empirical investigation, confirm that adaptive adversaries pose a serious threat to black-box AI-based systems, rekindling the proverbial arms race. Notably, our approach outperforms the state-of-the-art black-box attacks and defenses, while bringing them together to render effective insights into the robustness of real-world deployed ML-based systems.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 38 of its 64 references that are in the map. First posted on arXiv 20 Dec 2023.

Builds on

38 of 64 references.

References · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  4. 2016Concrete Problems in AI Safetycited by 1,582
  5. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  6. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  7. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  8. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  9. 2017Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Modelscited by 459
  10. 2019HopSkipJumpAttack: A Query-Efficient Decision-Based Attackcited by 328
  11. 2015Strategic Classificationcited by 256
  12. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209
26 more
  1. 2018PRADA: Protecting against DNN Model Stealing Attackscited by 159
  2. 2021On the Convergence and Robustness of Adversarial Trainingcited by 127
  3. 2017Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problemscited by 126
  4. 2019Sign-OPT: A Query-Efficient Hard-label Adversarial Attackcited by 118
  5. 2020QEBA: Query-Efficient Boundary-Based Blackbox Attackcited by 103
  6. 2017A Survey of Learning in Multiagent Environments: Dealing with Non-Stationaritycited by 93
  7. 2019Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learningcited by 68
  8. 2018Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learningcited by 60
  9. 2020SurFree: a fast surrogate-free black-box attackcited by 59
  10. 2020Functionality-preserving Black-box Optimization of Adversarial Windows Malwarecited by 58
  11. 2019Stateful Detection of Black-Box Adversarial Attackscited by 55
  12. 2022"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicecited by 55
  13. 2020Transcending Transcend: Revisiting Malware Classification in the Presence of Concept Driftcited by 53
  14. 2018Guessing Smart: Biased Sampling for Efficient Black-Box Adversarial Attackscited by 46
  15. 2021Random Noise Defense Against Query-Based Black-Box Attackscited by 46
  16. 2021AutoDropout: Learning Dropout Patterns to Regularize Deep Networkscited by 35
  17. 2020Blacklight: Scalable Defense for Neural Networks against Query-Based Black-Box Attackscited by 34
  18. 2020Adversarial Example Gamescited by 34
  19. 2020Approximate exploitability: Learning a best response in large gamescited by 23
  20. 2021Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examplescited by 23
  21. 2020A Game Theoretic Analysis of Additive Adversarial Attacks and Defensescited by 21
  22. 2021On the Effectiveness of Small Input Noise for Defending Against Query-based Black-Box Attackscited by 17
  23. 2020Multi-agent Reinforcement Learning in Bayesian Stackelberg Markov Games for Adaptive Moving Target Defensecited by 14
  24. 2023Evading Black-box Classifiers Without Breaking Eggscited by 13
  25. 2023Stateful Defenses for Machine Learning Models Are Not Yet Secure Against Black-box Attackscited by 10
  26. 2022Preprocessors Matter! Realistic Decision-Based Attacks on Machine Learning Systemscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  2. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  3. 2026SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecuritycited by 4
  4. 2026REPLICANT: Learning Policies for Evading and Hardening Malware Detectorscited by 0