mlsanityData to 7 Oct 2026

Releasing Inequality Phenomenon in ℓ∞-norm Adversarial Training via Input Gradient Distillation

arXiv 2305.09305 ↗First posted 16 May 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Adversarial training (AT) is considered the most effective defense against adversarial attacks. However, a recent study revealed that (ℓ∞)-norm adversarial training ((ℓ∞)-AT) will also induce unevenly distributed input gradients, which is called the inequality phenomenon. This phenomenon makes the (ℓ∞)-norm adversarially trained model more vulnerable than the standard-trained model when high-attribution or randomly selected pixels are perturbed, enabling robust and practical black-box attacks against (ℓ∞)-adversarially trained models. In this paper, we propose a simple yet effective method called Input Gradient Distillation (IGD) to release the inequality phenomenon in ℓ∞-AT. IGD distills the standard-trained teacher model's equal decision pattern into the ℓ∞-adversarially trained student model by aligning input gradients of the student model and the standard-trained model with the Cosine Similarity. Experiments show that IGD can mitigate the inequality phenomenon and its threats while preserving adversarial robustness. Compared to vanilla ℓ∞-AT, IGD reduces error rates against inductive noise, inductive occlusion, random noise, and noisy images in ImageNet-C by up to 60%, 16%, 50%, and 21%, respectively. Other than empirical experiments, we also conduct a theoretical analysis to explain why releasing the inequality phenomenon can improve such robustness and discuss why the severity of the inequality phenomenon varies according to the dataset's image resolution. Our code is available at https://github.com/fhdnskfbeuv/Inuput-Gradient-Distillation

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

27 of 46 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  7. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  8. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  9. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  10. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  11. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  12. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
15 more
  1. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  2. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 295
  3. 2019A Fourier Perspective on Model Robustness in Computer Visioncited by 288
  4. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  5. 2020Understanding and Improving Fast Adversarial Trainingcited by 213
  6. 2018MMA Training: Direct Input Space Margin Maximization through Adversarial Trainingcited by 171
  7. 2022Evaluating the Adversarial Robustness of Adaptive Test-time Defensescited by 64
  8. 2019Scaleable input gradient regularization for adversarial robustnesscited by 45
  9. 2019What it Thinks is Important is Important: Robustness Transfers through Input Gradientscited by 34
  10. 2019Detecting and Diagnosing Adversarial Images with Class-Conditional Capsule Reconstructionscited by 30
  11. 2021Mutual Adversarial Training: Learning together is better than going alonecited by 17
  12. 2020Connecting the Dots: Detecting Adversarial Perturbations Using Context Inconsistencycited by 16
  13. 2021How and When Adversarial Robustness Transfers in Knowledge Distillation?cited by 15
  14. 2022One-vs-the-Rest Loss to Focus on Important Samples in Adversarial Trainingcited by 7
  15. 2021Push Stricter to Decide Better: A Class-Conditional Feature Adaptive Framework for Improving Adversarial Robustnesscited by 6

Built on by

Not cited yet.