mlsanityData to 14 Oct 2026

Auto-ART: Structured Literature Synthesis and Automated Adversarial Robustness Testing

arXiv 2604.20704 ↗First posted 22 Apr 2026cs.CR cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Adversarial robustness evaluation underpins every claim of trustworthy ML deployment, yet the field suffers from fragmented protocols and undetected gradient masking. We make two contributions. (1) Structured synthesis. We analyze nine peer-reviewed corpus sources (2020--2026) through seven complementary protocols, producing the first end-to-end structured analysis of the field's consensus and unresolved challenges. (2) Auto-ART framework. We introduce Auto-ART, an open-source framework that operationalizes identified gaps: 50+ attacks, 28 defense modules, the Robustness Diagnostic Index (RDI), and gradient-masking detection. It supports multi-norm evaluation (l1/l2/linf/semantic/spatial) and compliance mapping to NIST AI RMF, OWASP LLM Top 10, and the EU AI Act. Empirical validation on RobustBench demonstrates that Auto-ART's pre-screening identifies gradient masking in 92% of flagged cases, and RDI rankings correlate highly with full AutoAttack. Multi-norm evaluation exposes a 23.5 pp gap between average and worst-case robustness on state-of-the-art models. No prior work combines such structured meta-scientific analysis with an executable evaluation framework bridging literature gaps into engineering.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 60 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

33 of 60 references.

References · 33

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  5. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  6. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  9. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  10. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  11. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  12. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
21 more
  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  2. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 279
  3. 2018Adversarial Robustness Toolbox v1.0.0cited by 215
  4. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 182
  5. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
  6. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 120
  7. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 110
  8. 2020Opportunities and Challenges in Deep Learning Adversarial Robustness: A Surveycited by 72
  9. 2020Adversarial Machine Learning in Image Classification: A Survey Towards the Defender's Perspectivecited by 59
  10. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 43
  11. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objectivecited by 15
  12. 2025AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defensescited by 15
  13. 2020Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustnesscited by 11
  14. 2025Robustness and Cybersecurity in the EU Artificial Intelligence Actcited by 11
  15. 2023MultiRobustBench: Benchmarking Robustness Against Multiple Attackscited by 7
  16. 2024DiffBreak: Is Diffusion-Based Purification Robust?cited by 6
  17. 2024SoundnessBench: A Soundness Benchmark for Neural Network Verifierscited by 5
  18. 2025Adapting to Evolving Adversaries with Regularized Continual Robust Trainingcited by 2
  19. 2024Position: Towards Resilience Against Adversarial Examplescited by 1
  20. 2024Adversarial Robustness Overestimation and Instability in TRADEScited by 1
  21. 2025RDI: An adversarial robustness evaluation metric for deep neural networks based on model statistical featurescited by 1

Built on by

Not cited yet.