mlsanityData to 7 Oct 2026

AlphaNet: Improved Training of Supernets with Alpha-Divergence

arXiv 2102.07954 ↗First posted 16 Feb 2021cs.CV cs.AI stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

Weight-sharing neural architecture search (NAS) is an effective technique for automating efficient neural architecture design. Weight-sharing NAS builds a supernet that assembles all the architectures as its sub-networks and jointly trains the supernet with the sub-networks. The success of weight-sharing NAS heavily relies on distilling the knowledge of the supernet to the sub-networks. However, we find that the widely used distillation divergence, i.e., KL divergence, may lead to student sub-networks that over-estimate or under-estimate the uncertainty of the teacher supernet, leading to inferior performance of the sub-networks. In this work, we propose to improve the supernet training with a more generalized alpha-divergence. By adaptively selecting the alpha-divergence, we simultaneously prevent the over-estimation or under-estimation of the uncertainty of the teacher model. We apply the proposed alpha-divergence based supernets training to both slimmable neural networks and weight-sharing NAS, and demonstrate significant improvements. Specifically, our discovered model family, AlphaNet, outperforms prior-art models on a wide range of FLOPs regimes, including BigNAS, Once-for-All networks, and AttentiveNAS. We achieve ImageNet top-1 accuracy of 80.0% with only 444M FLOPs. Our code and pretrained models are available at https://github.com/facebookresearch/AlphaNet.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

41 of 53 references.

References · 41

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  7. 2019Searching for MobileNetV3cited by 1,984
  8. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  11. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  12. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
29 more
  1. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  2. 2020Designing Network Design Spacescited by 954
  3. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  4. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  5. 2019Relational Knowledge Distillationcited by 873
  6. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  7. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  8. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  9. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  10. 2019Variational Information Distillation for Knowledge Transfercited by 321
  11. 2018Slimmable Neural Networkscited by 301
  12. 2019NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detectioncited by 289
  13. 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
  14. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  15. 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 192
  16. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  17. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  18. 2019FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Searchcited by 129
  19. 2019Blockwisely Supervised Neural Architecture Search with Knowledge Distillationcited by 81
  20. 2020AttentiveNAS: Improving Neural Architecture Search via Attentive Samplingcited by 65
  21. 2019Densely Connected Search Space for More Flexible Neural Architecture Searchcited by 52
  22. 2019Learning What and Where to Transfercited by 43
  23. 2019Customizable Architecture Search for Semantic Segmentationcited by 41
  24. 2020Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Searchcited by 38
  25. 2020FBNetV3: Joint Architecture-Recipe Search using Predictor Pretrainingcited by 35
  26. 2020Distilling Optimal Neural Networks: Rapid Search in Diverse Spacescited by 32
  27. 2019Hint-Based Training for Non-Autoregressive Machine Translationcited by 28
  28. 2018Variational Inference with Tail-adaptive f-Divergencecited by 25
  29. 2019Overcoming Multi-Model Forgettingcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 9; 2024: 7; 2025: 5; 2026: 3.

36 papers.

Sort:

  1. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  2. 2022Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?cited by 32
  3. 2023Weight subcloning: direct initialization of transformers using larger pretrained onescited by 30
  4. 2021UniNet: Unified Architecture Search with Convolution, Transformer, and MLPcited by 23
  5. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 21
  6. 2022On Redundancy and Diversity in Cell-based Neural Architecture Searchcited by 18
  7. 2021FBNetV5: Neural Architecture Search for Multiple Tasks in One Runcited by 15
  8. 2024AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deploymentcited by 12
  9. 2023ElasticViT: Conflict-aware Supernet Training for Deploying Fast Vision Transformer on Diverse Mobile Devicescited by 10
  10. 2023EQ-Net: Elastic Quantization Neural Networkscited by 10
  11. 2023Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devicescited by 8
  12. 2022The Lottery Ticket Hypothesis for Self-attention in Convolutional Neural Networkcited by 7
24 more
  1. 2026Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspectivecited by 5
  2. 2022Transfer Learning with Pre-trained Conditional Generative Modelscited by 4
  3. 2022Supernet Training for Federated Image Classification under System Heterogeneitycited by 4
  4. 2022ObfuNAS: A Neural Architecture Search-based DNN Obfuscation Approachcited by 4
  5. 2023TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Modelscited by 4
  6. 2024DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisionscited by 4
  7. 2024P2-ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformercited by 4
  8. 2025Subnet-Aware Dynamic Supernet Training for Neural Architecture Searchcited by 4
  9. 2022A Survey of Supernet Optimization and its Applications: Spatial and Temporal Optimization for Neural Architecture Searchcited by 2
  10. 2023LISSNAS: Locality-based Iterative Search Space Shrinkage for Neural Architecture Searchcited by 2
  11. 2024When Training-Free NAS Meets Vision Transformer: A Neural Tangent Kernel Perspectivecited by 2
  12. 2022A Semi-Decoupled Approach to Fast and Optimal Hardware-Software Co-Design of Neural Acceleratorscited by 1
  13. 2023Adaptive Depth Networks with Skippable Sub-Pathscited by 1
  14. 2024SONATA: Self-adaptive Evolutionary Framework for Hardware-aware Neural Architecture Searchcited by 1
  15. 2024NASH: Neural Architecture and Accelerator Search for Multiplication-Reduced Hybrid Modelscited by 1
  16. 2022UniNet: Unified Architecture Search with Convolution, Transformer, and MLPcited by 0
  17. 2022Slimmable Pruned Neural Networkscited by 0
  18. 2023NetDistiller: Empowering Tiny Deep Learning via In-Situ Distillationcited by 0
  19. 2024NAS-BNN: Neural Architecture Search for Binary Neural Networkscited by 0
  20. 2025Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insightscited by 0
  21. 2025A Memory-Efficient Framework for Deformable Transformer with Neural Architecture Searchcited by 0
  22. 2025AutoTailor: Automatic and Efficient Adaptive Model Deployment for Diverse Edge Devicescited by 0
  23. 2026The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Workscited by 0
  24. 2026No Unique Minimizer, No Problem: On the Consistency of Robust Neural Classifierscited by 0