mlsanityData to 14 Oct 2026

Adversarial Training via Adaptive Knowledge Amalgamation of an Ensemble of Teachers

arXiv 2405.13324 ↗First posted 22 May 2024cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Adversarial training (AT) is a popular method for training robust deep neural networks (DNNs) against adversarial attacks. Yet, AT suffers from two shortcomings: (i) the robustness of DNNs trained by AT is highly intertwined with the size of the DNNs, posing challenges in achieving robustness in smaller models; and (ii) the adversarial samples employed during the AT process exhibit poor generalization, leaving DNNs vulnerable to unforeseen attack types. To address these dual challenges, this paper introduces adversarial training via adaptive knowledge amalgamation of an ensemble of teachers (AT-AKA). In particular, we generate a diverse set of adversarial samples as the inputs to an ensemble of teachers; and then, we adaptively amalgamate the logtis of these teachers to train a generalized-robust student. Through comprehensive experiments, we illustrate the superior efficacy of AT-AKA over existing AT methods and adversarial robustness distillation techniques against cutting-edge attacks, including AutoAttack.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 47 of its 63 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

47 of 63 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2016Wide Residual Networkscited by 3,070
  8. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  9. 2016Adversarial examples in the physical worldcited by 1,769
  10. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  11. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  12. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
35 more
  1. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
  2. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 567
  3. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  4. 2019Similarity-Preserving Knowledge Distillationcited by 516
  5. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 507
  6. 2020Overfitting in adversarially robust deep learningcited by 506
  7. 2017Defense against Adversarial Attacks Using High-Level Representation Guided Denoisercited by 329
  8. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  9. 2018Feature Denoising for Improving Adversarial Robustnesscited by 314
  10. 2018Model compression via distillation and quantizationcited by 289
  11. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 278
  12. 2017PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examplescited by 244
  13. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 231
  14. 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
  15. 2020Bag of Tricks for Adversarial Trainingcited by 200
  16. 2019Online Knowledge Distillation with Diverse Peerscited by 128
  17. 2019Adversarially Robust Distillationcited by 125
  18. 2018Towards the first adversarially robust neural network model on MNISTcited by 104
  19. 2019Adversarial Robustness Against the Union of Multiple Perturbation Modelscited by 88
  20. 2020Do Wider Neural Networks Really Help Adversarial Robustness?cited by 79
  21. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 79
  22. 2019Interpreting Adversarially Trained Convolutional Neural Networkscited by 75
  23. 2020DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensemblescited by 73
  24. 2019Improving Adversarial Robustness of Ensembles with Diversity Trainingcited by 69
  25. 2019The Limitations of Adversarial Training and the Blind-Spot Attackcited by 63
  26. 2020Peer Collaborative Learning for Online Knowledge Distillationcited by 58
  27. 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 56
  28. 2018Improving the Generalization of Adversarial Training with Domain Adaptationcited by 48
  29. 2021TRS: Transferability Reduced Ensemble via Encouraging Gradient Diversity and Model Smoothnesscited by 38
  30. 2021Distill on the Go: Online knowledge distillation in self-supervised learningcited by 22
  31. 2020Encoding Robustness to Image Style via Adversarial Feature Perturbationscited by 18
  32. 2024AdaFed: Fair Federated Learning via Adaptive Common Descent Directioncited by 13
  33. 2021Thundernna: a white box adversarial attackcited by 4
  34. 2022Bayesian Learning with Information Gain Provably Bounds Risk for a Robust Adversarial Defensecited by 4
  35. 2024Robustness Against Adversarial Attacks via Learning Confined Adversarial Polytopescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026ASMIL: Attention-Stabilized Multiple Instance Learning for Whole Slide Imagingcited by 6
  2. 2025Improving Open-Set Semantic Segmentation in 3D Point Clouds by Conditional Channel Capacity Maximization: Preliminary Resultscited by 0