# Auto-ART: Structured Literature Synthesis and Automated Adversarial Robustness Testing

arXiv: https://arxiv.org/abs/2604.20704 · HTML: https://mlsanity.com/paper/2604.20704/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 60 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Builds on (most cited first, 30 of 33)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [Recent Advances in Adversarial Training for Adversarial Robustness](https://mlsanity.com/paper/2102.01356/) (2021)
- [Adversarial Robustness Toolbox v1.0.0](https://mlsanity.com/paper/1807.01069/) (2018)
- [Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach](https://mlsanity.com/paper/1801.10578/) (2018)
- [AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents](https://mlsanity.com/paper/2406.13352/) (2024)
- [Dissecting Adversarial Robustness of Multimodal LM Agents](https://mlsanity.com/paper/2406.12814/) (2024)
- [Improved Techniques for Optimization-Based Jailbreaking on Large Language Models](https://mlsanity.com/paper/2405.21018/) (2024)
- [Opportunities and Challenges in Deep Learning Adversarial Robustness: A Survey](https://mlsanity.com/paper/2007.00753/) (2020)
- [Adversarial Machine Learning in Image Classification: A Survey Towards the Defender's Perspective](https://mlsanity.com/paper/2009.03728/) (2020)
- [AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration](https://mlsanity.com/paper/2503.15754/) (2025)
- [REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective](https://mlsanity.com/paper/2502.17254/) (2025)
- [AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defenses](https://mlsanity.com/paper/2503.01811/) (2025)
- [Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness](https://mlsanity.com/paper/2006.13726/) (2020)
- [Robustness and Cybersecurity in the EU Artificial Intelligence Act](https://mlsanity.com/paper/2502.16184/) (2025)
- [MultiRobustBench: Benchmarking Robustness Against Multiple Attacks](https://mlsanity.com/paper/2302.10980/) (2023)
- [DiffBreak: Is Diffusion-Based Purification Robust?](https://mlsanity.com/paper/2411.16598/) (2024)
- [SoundnessBench: A Soundness Benchmark for Neural Network Verifiers](https://mlsanity.com/paper/2412.03154/) (2024)
- [Adapting to Evolving Adversaries with Regularized Continual Robust Training](https://mlsanity.com/paper/2502.04248/) (2025)

## Abstract (arXiv)

Adversarial robustness evaluation underpins every claim of trustworthy ML deployment, yet the field suffers from fragmented protocols and undetected gradient masking. We make two contributions. (1) Structured synthesis. We analyze nine peer-reviewed corpus sources (2020--2026) through seven complementary protocols, producing the first end-to-end structured analysis of the field's consensus and unresolved challenges. (2) Auto-ART framework. We introduce Auto-ART, an open-source framework that operationalizes identified gaps: 50+ attacks, 28 defense modules, the Robustness Diagnostic Index (RDI), and gradient-masking detection. It supports multi-norm evaluation (l1/l2/linf/semantic/spatial) and compliance mapping to NIST AI RMF, OWASP LLM Top 10, and the EU AI Act. Empirical validation on RobustBench demonstrates that Auto-ART's pre-screening identifies gradient masking in 92% of flagged cases, and RDI rankings correlate highly with full AutoAttack. Multi-norm evaluation exposes a 23.5 pp gap between average and worst-case robustness on state-of-the-art models. No prior work combines such structured meta-scientific analysis with an executable evaluation framework bridging literature gaps into engineering.