Indirect Gradient Matching for Adversarial Robust Distillation

arXiv 2312.03286 ↗First posted 6 Dec 2023cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Adversarial training significantly improves adversarial robustness, but superior performance is primarily attained with large models. This substantial performance gap for smaller models has spurred active research into adversarial distillation (AD) to mitigate the difference. Existing AD methods leverage the teacher's logits as a guide. In contrast to these approaches, we aim to transfer another piece of knowledge from the teacher, the input gradient. In this paper, we propose a distillation module termed Indirect Gradient Distillation Module (IGDM) that indirectly matches the student's input gradient with that of the teacher. Experimental results show that IGDM seamlessly integrates with existing AD methods, significantly enhancing their performance. Particularly, utilizing IGDM on the CIFAR-100 dataset improves the AutoAttack accuracy from 28.06% to 30.32% with the ResNet-18 architecture and from 26.18% to 29.32% with the MobileNetV2 architecture when integrated into the SOTA method without additional data augmentation.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 51 of its 68 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

51 of 68 references.

References · 51

2015Deep Residual Learning for Image Recognitioncited by 51,809
2015Distilling the Knowledge in a Neural Networkcited by 10,076
2014Explaining and Harnessing Adversarial Examplescited by 7,287
2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
2016Wide Residual Networkscited by 3,057
2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
2016Identity Mappings in Deep Residual Networkscited by 1,960
2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
2018Robustness May Be at Odds with Accuracycited by 873

38 more

2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
2019Unlabeled Data Improves Adversarial Robustnesscited by 415
2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 401
2019Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attackcited by 333
2018Feature Denoising for Improving Adversarial Robustnesscited by 312
2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 242
2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 201
2020Bag of Tricks for Adversarial Trainingcited by 199
2019Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systemscited by 151
2019Adversarial Robustness through Local Linearizationcited by 146
2021Focal and Global Knowledge Distillation for Detectorscited by 138
2017Sobolev Training for Neural Networkscited by 136
2019Adversarially Robust Distillationcited by 124
2022On Adversarial Robustness of Trajectory Prediction for Autonomous Vehiclescited by 89
2017Keeping the Bad Guys Out: Protecting and Vaccinating Deep Learning with JPEG Compressioncited by 86
2019Functional Adversarial Attackscited by 82
2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 77
2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 76
2021Improving Adversarial Robustness via Channel-wise Activation Suppressingcited by 76
2023Decoupled Kullback-Leibler Divergence Losscited by 58
2021Reliable Adversarial Distillation with Unreliable Teacherscited by 55
2023CFA: Class-wise Calibrated Fair Adversarial Trainingcited by 53
2023Data Augmentation Alone Can Improve Adversarial Trainingcited by 43
2021Consistency Regularization for Adversarial Robustnesscited by 42
2022Enhancing Adversarial Training with Second-Order Statistics of Weightscited by 42
2019Feature Fusion for Online Mutual Knowledge Distillationcited by 40
2023Does Physical Adversarial Example Really Matter to Autonomous Driving? Towards System-Level Effect of Adversarial Object Evasion Attackcited by 37
2019What it Thinks is Important is Important: Robustness Transfers through Input Gradientscited by 34
2023Randomized Adversarial Training via Taylor Expansioncited by 28
2023Improving Generalization of Adversarial Training via Robust Critical Fine-Tuningcited by 23
2023Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustnesscited by 20
2022On the benefits of knowledge distillation for adversarial robustnesscited by 17
2021LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Trainingcited by 13
2023Gradient-Guided Knowledge Distillation for Object Detectorscited by 11

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

2025Long-tailed Adversarial Training with Self-Distillationcited by 5
2024Enhancing Adversarial Robustness via Uncertainty-Aware Distributional Adversarial Trainingcited by 2
2026Fitting Is Not Enough: Smoothness in Extremely Quantized LLMscited by 1
2025Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillationcited by 0
2026Information Theoretic Adversarial Training of Large Language Modelscited by 0