mlsanityData to 14 Oct 2026

SCOOTER: A Human Evaluation Framework for Unrestricted Adversarial Examples

arXiv 2507.07776 ↗First posted 10 Jul 2025cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Unrestricted adversarial attacks aim to fool computer vision models without being constrained by ℓp-norm bounds to remain imperceptible to humans, for example, by changing an object's color. This allows attackers to circumvent traditional, norm-bounded defense strategies such as adversarial training or certified defense strategies. However, due to their unrestricted nature, there are also no guarantees of norm-based imperceptibility, necessitating human evaluations to verify just how authentic these adversarial examples look. While some related work assesses this vital quality of adversarial attacks, none provide statistically significant insights. This issue necessitates a unified framework that supports and streamlines such an assessment for evaluating and comparing unrestricted attacks. To close this gap, we introduce SCOOTER - an open-source, statistically powered framework for evaluating unrestricted adversarial examples. Our contributions are: (i) best-practice guidelines for crowd-study power, compensation, and Likert equivalence bounds to measure imperceptibility; (ii) the first large-scale human vs. model comparison across 346 human participants showing that three color-space attacks and three diffusion-based attacks fail to produce imperceptible images. Furthermore, we found that GPT-4o can serve as a preliminary test for imperceptibility, but it only consistently detects adversarial examples for four out of six tested attacks; (iii) open-source software tools, including a browser-based task template to collect annotations and analysis scripts in Python and R; (iv) an ImageNet-derived benchmark dataset containing 3K real images, 7K adversarial examples, and over 34K human ratings. Our findings demonstrate that automated vision systems do not align with human perception, reinforcing the need for a ground-truth SCOOTER benchmark.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 57 of its 97 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

57 of 97 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  10. 2013Intriguing properties of neural networkscited by 5,155
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
38 more
  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  3. 2018Demystifying MMD GANscited by 1,293
  4. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  5. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  6. 2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 486
  7. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 434
  8. 2018Assessing Generative Models via Precision and Recallcited by 406
  9. 2020Are we done with ImageNet?cited by 333
  10. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
  11. 2023TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessmentcited by 226
  12. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  13. 2016A study of the effect of JPG compression on adversarial imagescited by 197
  14. 2020Perceptual Adversarial Robustness: Defense Against Unseen Threat Modelscited by 137
  15. 2018Constructing Unrestricted Adversarial Examples with Generative Modelscited by 135
  16. 2020SoK: Certified Robustness for Deep Neural Networkscited by 105
  17. 2018Adversarial Attacks and Defences Competitioncited by 96
  18. 2018SparseFool: a few pixels make a big differencecited by 94
  19. 2023Diffusion Models for Imperceptible and Transferable Adversarial Attackcited by 94
  20. 2019SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editingcited by 93
  21. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90
  22. 2019Wasserstein Adversarial Examples via Projected Sinkhorn Iterationscited by 89
  23. 2019ColorFool: Semantic Adversarial Colorizationcited by 89
  24. 2018Semantic Adversarial Examplescited by 86
  25. 2017A General Framework for Adversarial Examples with Objectivescited by 84
  26. 2019Unrestricted Adversarial Examples via Semantic Manipulationcited by 84
  27. 2018Adversarial Examples that Fool both Computer Vision and Time-Limited Humanscited by 73
  28. 2019Towards Large yet Imperceptible Adversarial Image Perturbations with Perceptual Color Distancecited by 73
  29. 2023Content-based Unrestricted Adversarial Attackcited by 73
  30. 2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 68
  31. 2023Diffusion-Based Adversarial Sample Generation for Improved Stealthiness and Controllabilitycited by 66
  32. 2019HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Modelscited by 59
  33. 2019Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifierscited by 57
  34. 2023AdvDiff: Generating Unrestricted Adversarial Examples using Diffusion Modelscited by 46
  35. 2018Unrestricted Adversarial Examplescited by 40
  36. 2018Attacks on State-of-the-Art Face Recognition using Attentional Adversarial Attack Generative Networkcited by 38
  37. 2022Natural Color Fool: Towards Boosting Black-box Unrestricted Attackscited by 38
  38. 2023DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purificationcited by 36

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoderscited by 0