mlsanityData to 14 Oct 2026

MMARD: Improving the Min-Max Optimization Process in Adversarial Robustness Distillation

arXiv 2503.06559 ↗First posted 9 Mar 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Adversarial Robustness Distillation (ARD) is a promising task to boost the robustness of small-capacity models with the guidance of the pre-trained robust teacher. The ARD can be summarized as a min-max optimization process, i.e., synthesizing adversarial examples (inner) & training the student (outer). Although competitive robustness performance, existing ARD methods still have issues. In the inner process, the synthetic training examples are far from the teacher's decision boundary leading to important robust information missing. In the outer process, the student model is decoupled from learning natural and robust scenarios, leading to the robustness saturation, i.e., student performance is highly susceptible to customized teacher selection. To tackle these issues, this paper proposes a general Min-Max optimization Adversarial Robustness Distillation (MMARD) method. For the inner process, we introduce the teacher's robust predictions, which drive the training examples closer to the teacher's decision boundary to explore more robust knowledge. For the outer process, we propose a structured information modeling method based on triangular relationships to measure the mutual information of the model in natural and robust scenarios and enhance the model's ability to understand multi-scenario mapping relationships. Experiments show our MMARD achieves state-of-the-art performance on multiple benchmarks. Besides, MMARD is plug-and-play and convenient to combine with existing methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

38 of 50 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  10. 2013Intriguing properties of neural networkscited by 5,155
  11. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  12. 2016Wide Residual Networkscited by 3,070
26 more
  1. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  2. 2016Adversarial examples in the physical worldcited by 1,769
  3. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  4. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  5. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  6. 2019When Does Label Smoothing Help?cited by 988
  7. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  8. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  9. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 278
  10. 2019Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systemscited by 153
  11. 2021On the Convergence and Robustness of Adversarial Trainingcited by 130
  12. 2019Adversarially Robust Distillationcited by 125
  13. 2020Adversarial Machine Learning -- Industry Perspectivescited by 122
  14. 2018ComDefend: An Efficient Image Compression Model to Defend Adversarial Examplescited by 105
  15. 2019When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attackscited by 87
  16. 2020Do Wider Neural Networks Really Help Adversarial Robustness?cited by 79
  17. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 79
  18. 2021Improving Adversarial Robustness via Channel-wise Activation Suppressingcited by 76
  19. 20203D Point Cloud Processing and Learning for Autonomous Drivingcited by 68
  20. 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 56
  21. 2018Knowledge Distillation with Adversarial Samples Supporting Decision Boundarycited by 49
  22. 2018Self-supervised Knowledge Distillation Using Singular Value Decompositioncited by 48
  23. 2022Towards Practical Certifiable Patch Defense with Vision Transformercited by 44
  24. 2019Characterizing the Decision Boundary of Deep Neural Networkscited by 32
  25. 2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Againcited by 23
  26. 2022Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classificationcited by 19

Built on by

Not cited yet.