# Adversarial Risk and the Dangers of Evaluating Against Weak Attacks

arXiv: https://arxiv.org/abs/1802.05666 · HTML: https://mlsanity.com/paper/1802.05666/

## Key facts

- Cited by 221 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2018.

## Built on by (most cited first, 30 of 221)

- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Certifying LLM Safety against Adversarial Prompting](https://mlsanity.com/paper/2309.02705/) (2023)
- [Improving Robustness using Generated Data](https://mlsanity.com/paper/2110.09468/) (2021)
- [Fixing Data Augmentation to Improve Adversarial Robustness](https://mlsanity.com/paper/2103.01946/) (2021)
- [Data Augmentation Can Improve Robustness](https://mlsanity.com/paper/2111.05328/) (2021)
- [Adversarial purification with Score-based generative models](https://mlsanity.com/paper/2106.06041/) (2021)
- [(Certified!!) Adversarial Robustness for Free!](https://mlsanity.com/paper/2206.10550/) (2022)
- [Feature Importance-aware Transferable Adversarial Attacks](https://mlsanity.com/paper/2107.14185/) (2021)
- [Anti-DreamBooth: Protecting users from personalized text-to-image synthesis](https://mlsanity.com/paper/2303.15433/) (2023)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [Improving Adversarial Transferability via Neuron Attribution-Based Attacks](https://mlsanity.com/paper/2204.00008/) (2022)
- [Guided Diffusion Model for Adversarial Purification](https://mlsanity.com/paper/2205.14969/) (2022)
- [RobustART: Benchmarking Robustness on Architecture Design and Training Techniques](https://mlsanity.com/paper/2109.05211/) (2021)
- [Evaluating the Adversarial Robustness of Adaptive Test-time Defenses](https://mlsanity.com/paper/2202.13711/) (2022)
- [Boosting Adversarial Transferability by Block Shuffle and Rotation](https://mlsanity.com/paper/2308.10299/) (2023)
- [Structure Invariant Transformation for better Adversarial Transferability](https://mlsanity.com/paper/2309.14700/) (2023)
- [Adversarial Training for High-Stakes Reliability](https://mlsanity.com/paper/2205.01663/) (2022)
- [On Evaluating the Durability of Safeguards for Open-Weight LLMs](https://mlsanity.com/paper/2412.07097/) (2024)
- [Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraints](https://mlsanity.com/paper/2102.12827/) (2021)
- [PAL: Proxy-Guided Black-Box Attack on Large Language Models](https://mlsanity.com/paper/2402.09674/) (2024)
- [Improved Branch and Bound for Neural Network Verification via Lagrangian Decomposition](https://mlsanity.com/paper/2104.06718/) (2021)
- [Practical Evaluation of Adversarial Robustness via Adaptive Auto Attack](https://mlsanity.com/paper/2203.05154/) (2022)
- [A Comprehensive Evaluation Framework for Deep Model Robustness](https://mlsanity.com/paper/2101.09617/) (2021)
- [LiBRe: A Practical Bayesian Approach to Adversarial Detection](https://mlsanity.com/paper/2103.14835/) (2021)
- [Harnessing Perceptual Adversarial Patches for Crowd Counting](https://mlsanity.com/paper/2109.07986/) (2021)
- [Query-Efficient Black-box Adversarial Attacks Guided by a Transfer-based Prior](https://mlsanity.com/paper/2203.06560/) (2022)
- [DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification](https://mlsanity.com/paper/2311.16124/) (2023)
- [A Little Robustness Goes a Long Way: Leveraging Robust Features for Targeted Transfer Attacks](https://mlsanity.com/paper/2106.02105/) (2021)
- [Exploring Misclassifications of Robust Neural Networks to Enhance Adversarial Attacks](https://mlsanity.com/paper/2105.10304/) (2021)
- [The Many Faces of Adversarial Risk](https://mlsanity.com/paper/2201.08956/) (2022)

## Abstract (arXiv)

This paper investigates recently proposed approaches for defending against adversarial examples and evaluating adversarial robustness. We motivate 'adversarial risk' as an objective for achieving models robust to worst-case inputs. We then frame commonly used attacks and evaluation metrics as defining a tractable surrogate objective to the true adversarial risk. This suggests that models may optimize this surrogate rather than the true adversarial risk. We formalize this notion as 'obscurity to an adversary,' and develop tools and heuristics for identifying obscured models and designing transparent models. We demonstrate that this is a significant problem in practice by repurposing gradient-free optimization techniques into adversarial attacks, which we use to decrease the accuracy of several recently proposed defenses to near zero. Our hope is that our formulations and results will help researchers to develop more powerful defenses.