Revisiting Architecture-aware Knowledge Distillation: Smaller Models and Faster Search
Not on the map. Why?
Abstract (arXiv)
Knowledge Distillation (KD) has recently emerged as a popular method for compressing neural networks. In recent studies, generalized distillation methods that find parameters and architectures of student models at the same time have been proposed. Still, this search method requires a lot of computation to search for architectures and has the disadvantage of considering only convolutional blocks in their search space. This paper introduces a new algorithm, coined as Trust Region Aware architecture search to Distill knowledge Effectively (TRADE), that rapidly finds effective student architectures from several state-of-the-art architectures using trust region Bayesian optimization approach. Experimental results show our proposed TRADE algorithm consistently outperforms both the conventional NAS approach and pre-defined architecture under KD training.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 57 of its 69 references that are in the map. First posted on arXiv 27 Jun 2022.
Builds on
57 of 69 references.
References · 57
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2016Densely Connected Convolutional Networkscited by 6,691
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
- 2015SSD: Single Shot MultiBox Detectorcited by 3,125
- 2016Wide Residual Networkscited by 3,057
- 2018DARTS: Differentiable Architecture Searchcited by 1,782
38 more
- 2014FitNets: Hints for Thin Deep Netscited by 1,619
- 2020Knowledge Distillation: A Surveycited by 1,578
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
- 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
- 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
- 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
- 2020Designing Network Design Spacescited by 954
- 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
- 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
- 2019Relational Knowledge Distillationcited by 873
- 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
- 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
- 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
- 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
- 2021Early Convolutions Help Transformers See Bettercited by 395
- 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
- 2017Large-Scale Evolution of Image Classifierscited by 348
- 2019A Comprehensive Overhaul of Feature Distillationcited by 341
- 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
- 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
- 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 192
- 2017Hierarchical Representations for Efficient Architecture Searchcited by 188
- 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
- 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 175
- 2016Deep Pyramidal Residual Networkscited by 163
- 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 146
- 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
- 2019Orthogonal Convolutional Neural Networkscited by 122
- 2019Evaluating the Search Phase of Neural Architecture Searchcited by 120
- 2020Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Searchcited by 87
- 2018Knowledge Transfer with Jacobian Matchingcited by 73
- 2020AttentiveNAS: Improving Neural Architecture Search via Attentive Samplingcited by 65
- 2018Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?cited by 62
- 2017All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulationcited by 58
- 2020Few-shot Neural Architecture Searchcited by 50
- 2020Deep Isometric Learning for Visual Recognitioncited by 38
- 2020Distilling Optimal Neural Networks: Rapid Search in Diverse Spacescited by 32
- 2020Accelerator-aware Neural Network Design using AutoMLcited by 30
50 most cited of 57 All 57 (JSON)
Built on by
Citing papers per year: 2023: 2.
2 papers.