mlsanityData to 14 Oct 2026

Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation

arXiv 2512.10275 ↗First posted 11 Dec 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Adversarial distillation in the standard min-max adversarial training framework aims to transfer adversarial robustness from a large, robust teacher network to a compact student. However, existing work often neglects to incorporate state-of-the-art robust teachers. Through extensive analysis, we find that stronger teachers do not necessarily yield more robust students-a phenomenon known as robust saturation. While typically attributed to capacity gaps, we show that such explanations are incomplete. Instead, we identify adversarial transferability-the fraction of student-crafted adversarial examples that remain effective against the teacher-as a key factor in successful robustness transfer. Based on this insight, we propose Sample-wise Adaptive Adversarial Distillation (SAAD), which reweights training examples by their measured transferability without incurring additional computational cost. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet show that SAAD consistently improves AutoAttack robustness over prior methods. Our code is available at https://github.com/HongsinLee/saad.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 82 of its 92 references that are in the map. First posted on arXiv 11 Dec 2025.

Builds on

82 of 92 references.

References · 82

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  8. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  9. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  10. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  11. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  12. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
38 more
  1. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  2. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  3. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  4. 2018Robustness May Be at Odds with Accuracycited by 880
  5. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 685
  6. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  7. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
  8. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 446
  9. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
  10. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 403
  11. 2019HopSkipJumpAttack: A Query-Efficient Decision-Based Attackcited by 333
  12. 2018Adversarially Robust Generalization Requires More Datacited by 318
  13. 2018Feature Denoising for Improving Adversarial Robustnesscited by 314
  14. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  15. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 278
  16. 2021Improving Robustness using Generated Datacited by 234
  17. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  18. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 208
  19. 2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 204
  20. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  21. 2017The Space of Transferable Adversarial Examplescited by 158
  22. 2019Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systemscited by 153
  23. 2019Adversarial Robustness through Local Linearizationcited by 146
  24. 2019Adversarially Robust Distillationcited by 125
  25. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 119
  26. 2021Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?cited by 113
  27. 2020RayS: A Ray Searching Method for Hard-label Adversarial Attackcited by 96
  28. 2022On Adversarial Robustness of Trajectory Prediction for Autonomous Vehiclescited by 90
  29. 2017Keeping the Bad Guys Out: Protecting and Vaccinating Deep Learning with JPEG Compressioncited by 87
  30. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 79
  31. 2021Improving Adversarial Robustness via Channel-wise Activation Suppressingcited by 76
  32. 2021Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networkscited by 74
  33. 2019Adversarially robust transfer learningcited by 67
  34. 2023Decoupled Kullback-Leibler Divergence Losscited by 58
  35. 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 56
  36. 2023CFA: Class-wise Calibrated Fair Adversarial Trainingcited by 53
  37. 2020Precise Statistical Analysis of Classification Accuracies for Adversarial Trainingcited by 50
  38. 2021Consistency Regularization for Adversarial Robustnesscited by 42

50 most cited of 82 All 82 (JSON)

Built on by

Not cited yet.