# Adversarial Fooling Beyond "Flipping the Label"

arXiv: https://arxiv.org/abs/2004.12771 · HTML: https://mlsanity.com/paper/2004.12771/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2020.

## Built on by (most cited first, 5 of 5)

- [How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses](https://mlsanity.com/paper/2305.10862/) (2023)
- [Investigating Top-k White-Box and Transferable Black-box Attack](https://mlsanity.com/paper/2204.00089/) (2022)
- [LiSA: Leveraging Link Recommender to Attack Graph Neural Networks via Subgraph Injection](https://mlsanity.com/paper/2502.09271/) (2025)
- [Semantic Depth Matters: Explaining Errors of Deep Vision Networks through Perceived Class Similarities](https://mlsanity.com/paper/2504.09956/) (2025)
- [Semantically Guided Adversarial Testing of Vision Models Using Language Models](https://mlsanity.com/paper/2508.11341/) (2025)

## Abstract (arXiv)

Recent advancements in CNNs have shown remarkable achievements in various CV/AI applications. Though CNNs show near human or better than human performance in many critical tasks, they are quite vulnerable to adversarial attacks. These attacks are potentially dangerous in real-life deployments. Though there have been many adversarial attacks proposed in recent years, there is no proper way of quantifying the effectiveness of these attacks. As of today, mere fooling rate is used for measuring the susceptibility of the models, or the effectiveness of adversarial attacks. Fooling rate just considers label flipping and does not consider the cost of such flipping, for instance, in some deployments, flipping between two species of dogs may not be as severe as confusing a dog category with that of a vehicle. Therefore, the metric to quantify the vulnerability of the models should capture the severity of the flipping as well. In this work we first bring out the drawbacks of the existing evaluation and propose novel metrics to capture various aspects of the fooling. Further, for the first time, we present a comprehensive analysis of several important adversarial attacks over a set of distinct CNN architectures. We believe that the presented analysis brings valuable insights about the current adversarial attacks and the CNN models.