# Gradient Masking Causes CLEVER to Overestimate Adversarial Perturbation Size

arXiv: https://arxiv.org/abs/1804.07870 · HTML: https://mlsanity.com/paper/1804.07870/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2018.

## Built on by (most cited first, 9 of 9)

- [Robust Machine Learning Systems: Challenges, Current Trends, Perspectives, and the Road Ahead](https://mlsanity.com/paper/2101.02559/) (2021)
- [Adversarial Robustness Overestimation and Instability in TRADES](https://mlsanity.com/paper/2410.07675/) (2024)
- [Rethinking Robustness in Machine Learning: A Posterior Agreement Approach](https://mlsanity.com/paper/2503.16271/) (2025)
- [humancompatible.interconnect: Testing Properties of Repeated Uses of Interconnections of AI Systems](https://mlsanity.com/paper/2507.09626/) (2025)
- [Theoretical Analysis of Relative Errors in Gradient Computations for Adversarial Attacks with CE Loss](https://mlsanity.com/paper/2507.22428/) (2025)
- [Stochastic Sample Approximations of (Local) Moduli of Continuity](https://mlsanity.com/paper/2509.15368/) (2025)
- [Evaluating Adversarial Robustness with Expected Viable Performance](https://mlsanity.com/paper/2309.09928/) (2023)
- [A practical approach to evaluating the adversarial distance for machine learning classifiers](https://mlsanity.com/paper/2409.03598/) (2024)
- [Dummy-Aware Weighted Attack (DAWA): Breaking the Safe Sink in Dummy Class Defenses](https://mlsanity.com/paper/2603.29182/) (2026)

## Abstract (arXiv)

A key problem in research on adversarial examples is that vulnerability to adversarial examples is usually measured by running attack algorithms. Because the attack algorithms are not optimal, the attack algorithms are prone to overestimating the size of perturbation needed to fool the target model. In other words, the attack-based methodology provides an upper-bound on the size of a perturbation that will fool the model, but security guarantees require a lower bound. CLEVER is a proposed scoring method to estimate a lower bound. Unfortunately, an estimate of a bound is not a bound. In this report, we show that gradient masking, a common problem that causes attack methodologies to provide only a very loose upper bound, causes CLEVER to overestimate the size of perturbation needed to fool the model. In other words, CLEVER does not resolve the key problem with the attack-based methodology, because it fails to provide a lower bound.