RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation
Not on the map. Why?
Abstract (arXiv)
Deep Neural Networks are vulnerable to adversarial attacks. Neural Architecture Search (NAS), one of the driving tools of deep neural networks, demonstrates superior performance in prediction accuracy in various machine learning applications. However, it is unclear how it performs against adversarial attacks. Given the presence of a robust teacher, it would be interesting to investigate if NAS would produce robust neural architecture by inheriting robustness from the teacher. In this paper, we propose Robust Neural Architecture Search by Cross-Layer Knowledge Distillation (RNAS-CL), a novel NAS algorithm that improves the robustness of NAS by learning from a robust teacher through cross-layer knowledge distillation. Unlike previous knowledge distillation methods that encourage close student/teacher output only in the last layer, RNAS-CL automatically searches for the best teacher layer to supervise each student layer. Experimental result evidences the effectiveness of RNAS-CL and shows that RNAS-CL produces small and robust neural architecture.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 66 of its 77 references that are in the map. First posted on arXiv 19 Jan 2023.
Builds on
66 of 77 references.
References · 66
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2013Intriguing properties of neural networkscited by 5,114
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
- 2018DARTS: Differentiable Architecture Searchcited by 1,782
- 2014FitNets: Hints for Thin Deep Netscited by 1,619
- 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
38 more
- 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
- 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
- 2019Contrastive Multiview Codingcited by 1,297
- 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
- 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
- 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
- 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
- 2019Relational Knowledge Distillationcited by 873
- 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
- 2019Contrastive Representation Distillationcited by 734
- 2020Fast is better than free: Revisiting adversarial trainingcited by 723
- 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
- 2019Adversarial Training for Free!cited by 638
- 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
- 2017Countering Adversarial Images using Input Transformationscited by 517
- 2019Similarity-Preserving Knowledge Distillationcited by 507
- 2020Overfitting in adversarially robust deep learningcited by 504
- 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
- 2017Large-Scale Evolution of Image Classifierscited by 348
- 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
- 2019Variational Information Distillation for Knowledge Transfercited by 321
- 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
- 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
- 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
- 2018Adversarial Logit Pairingcited by 224
- 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
- 2016A study of the effect of JPG compression on adversarial imagescited by 196
- 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
- 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 182
- 2017Towards Robust Neural Networks via Random Self-ensemblecited by 158
- 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
- 2018Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Modelscited by 135
- 2019Adversarially Robust Distillationcited by 124
- 2021Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?cited by 113
- 2019Adversarial Robustness vs Model Compression, or Both?cited by 98
- 2018Adversarial Attacks and Defences Competitioncited by 96
- 2019When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attackscited by 87
- 2019Blockwisely Supervised Neural Architecture Search with Knowledge Distillationcited by 81
50 most cited of 66 All 66 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.