mlsanityData to 14 Oct 2026

Learn from the Past: A Proxy Guided Adversarial Defense Framework with Self Distillation Regularization

arXiv 2310.12713 ↗First posted 19 Oct 2023cs.LGcited by 0

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Adversarial Training (AT), pivotal in fortifying the robustness of deep learning models, is extensively adopted in practical applications. However, prevailing AT methods, relying on direct iterative updates for target model's defense, frequently encounter obstacles such as unstable training and catastrophic overfitting. In this context, our work illuminates the potential of leveraging the target model's historical states as a proxy to provide effective initialization and defense prior, which results in a general proxy guided defense framework, `LAST' (bf Learn from the Pbf ast). Specifically, LAST derives response of the proxy model as dynamically learned fast weights, which continuously corrects the update direction of the target model. Besides, we introduce a self-distillation regularized defense objective, ingeniously designed to steer the proxy model's update trajectory without resorting to external teacher models, thereby ameliorating the impact of catastrophic overfitting on performance. Extensive experiments and ablation studies showcase the framework's efficacy in markedly improving model robustness (e.g., up to 9.2% and 20.3% enhancement in robust accuracy on CIFAR10 and CIFAR100 datasets, respectively) and training stability. These improvements are consistently observed across various model architectures, larger datasets, perturbation sizes, and attack modalities, affirming LAST's ability to consistently refine both single-step and multi-step AT strategies. The code will be available at~urlhttps://github.com/callous-youth/LAST.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 44 of its 59 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

44 of 59 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  6. 2016Wide Residual Networkscited by 3,070
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  8. 2016Adversarial examples in the physical worldcited by 1,769
  9. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  10. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  11. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  12. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
32 more
  1. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  3. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  4. 2019Adversarial Training for Free!cited by 643
  5. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 567
  6. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 546
  7. 2020Overfitting in adversarially robust deep learningcited by 506
  8. 2017Defense against Adversarial Attacks Using High-Level Representation Guided Denoisercited by 329
  9. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 278
  10. 2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 243
  11. 2020Understanding and Improving Fast Adversarial Trainingcited by 214
  12. 2020Bag of Tricks for Adversarial Trainingcited by 200
  13. 2022(Certified!!) Adversarial Robustness for Free!cited by 132
  14. 2022Robustness and Accuracy Could Be Reconcilable by (Proper) Definitioncited by 111
  15. 2022LAS-AT: Adversarial Training with Learnable Attack Strategycited by 95
  16. 2018There Are Many Consistent Explanations of Unlabeled Data: Why You Should Averagecited by 82
  17. 2020Adversarial Distributional Training for Robust Deep Learningcited by 79
  18. 2020Understanding Catastrophic Overfitting in Single-step Adversarial Trainingcited by 76
  19. 2021Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimizationcited by 76
  20. 2020On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Themcited by 59
  21. 2022T-SEA: Transfer-based Self-Ensemble Attack on Object Detectioncited by 59
  22. 2021Exploring Memorization in Adversarial Trainingcited by 55
  23. 2021Meta Gradient Adversarial Attackcited by 55
  24. 2018Training verified learners with learned verifierscited by 50
  25. 2019SWALP : Stochastic Weight Averaging in Low-Precision Trainingcited by 42
  26. 2023Exploring and Exploiting Decision Boundary Dynamics for Adversarial Robustnesscited by 36
  27. 2019What it Thinks is Important is Important: Robustness Transfers through Input Gradientscited by 34
  28. 2020Towards Understanding Fast Adversarial Trainingcited by 33
  29. 2022Revisiting adapters with adversarial trainingcited by 18
  30. 2021Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overfittingcited by 14
  31. 2020Retrospective Loss: Looking Back to Improve Training of Deep Neural Networkscited by 3
  32. 2023PEARL: Preprocessing Enhanced Adversarial Robust Learning of Image Deraining for Semantic Segmentationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Keep on Swimming: Real Attackers Only Need Partial Knowledge of a Multi-Model System
  2. 2021Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?
  3. 2026Proxy Robustness in Vision Language Models is Effortlessly Transferable
  4. 2024A Novel Approach to Guard from Adversarial Attacks using Stable Diffusion
  5. 2026Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogates
  6. 2023Annealing Self-Distillation Rectification Improves Adversarial Training
  7. 2025Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models
  8. 2022Vanilla Feature Distillation for Improving the Accuracy-Robustness Trade-Off in Adversarial Training