mlsanityData to 7 Oct 2026

BN-NAS: Neural Architecture Search with Batch Normalization

arXiv 2108.07375 ↗First posted 16 Aug 2021cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

We present BN-NAS, neural architecture search with Batch Normalization (BN-NAS), to accelerate neural architecture search (NAS). BN-NAS can significantly reduce the time required by model training and evaluation in NAS. Specifically, for fast evaluation, we propose a BN-based indicator for predicting subnet performance at a very early training stage. The BN-based indicator further facilitates us to improve the training efficiency by only training the BN parameters during the supernet training. This is based on our observation that training the whole supernet is not necessary while training only BN parameters accelerates network convergence for network architecture search. Extensive experiments show that our method can significantly shorten the time of training supernet by more than 10 times and shorten the time of evaluating subnets by more than 600,000 times without losing accuracy.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 33 of its 37 references that are in the map. First posted on arXiv 16 Aug 2021.

Builds on

33 of 37 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  6. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  7. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  8. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  9. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  10. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  11. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  12. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
21 more
  1. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  2. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  3. 2018SNAS: Stochastic Neural Architecture Searchcited by 288
  4. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  5. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  6. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  7. 2019FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Searchcited by 129
  8. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 103
  9. 2018Searching for Efficient Multi-Scale Architectures for Dense Image Predictioncited by 96
  10. 2017Neural Optimizer Search with Reinforcement Learningcited by 82
  11. 2019DetNAS: Backbone Search for Object Detectioncited by 80
  12. 2019CARS: Continuous Evolution for Efficient Neural Architecture Searchcited by 79
  13. 2020Operation-Aware Soft Channel Pruning using Differentiable Maskscited by 74
  14. 2020GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernetcited by 69
  15. 2020EcoNAS: Finding Proxies for Economical Neural Architecture Searchcited by 61
  16. 2021GLiT: Neural Architecture Search for Global and Local Image Transformercited by 53
  17. 2020Angle-based Search Space Shrinking for Neural Architecture Searchcited by 32
  18. 2019Improving One-shot NAS by Suppressing the Posterior Fadingcited by 28
  19. 2020Evolving Search Space for Neural Architecture Searchcited by 23
  20. 2019Computation Reallocation for Object Detectioncited by 17
  21. 2020Inception Convolution with Efficient Dilation Searchcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 5; 2024: 2; 2025: 3; 2026: 1.

21 papers.

Sort:

  1. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  2. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  3. 2022Mixed-Precision Neural Network Quantization via Learned Layer-wise Importancecited by 53
  4. 2022Neural Architecture Search for Spiking Neural Networkscited by 52
  5. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  6. 2024AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Searchcited by 25
  7. 2021Analyzing and Mitigating Interference in Neural Architecture Searchcited by 20
  8. 2023LightCTS: A Lightweight Framework for Correlated Time Series Forecastingcited by 18
  9. 2022Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafingcited by 10
  10. 2022FIT: A Metric for Model Sensitivitycited by 10
  11. 2023Stimulative Training++: Go Beyond The Performance Limits of Residual Networkscited by 6
  12. 2025From Tiny Machine Learning to Tiny Deep Learning: A Surveycited by 5
9 more
  1. 2022Differentiable Architecture Search with Random Featurescited by 4
  2. 2022When NAS Meets Trees: An Efficient Algorithm for Neural Architecture Searchcited by 3
  3. 2021Training BatchNorm Only in Neural Architecture Search and Beyondcited by 2
  4. 2025Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weightscited by 2
  5. 2024EM-DARTS: Hierarchical Differentiable Architecture Search for Eye Movement Recognitioncited by 1
  6. 2025Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvaturecited by 1
  7. 2021Improving Differentiable Architecture Search with a Generative Modelcited by 0
  8. 2023Dynamical Isometry based Rigorous Fair Neural Architecture Searchcited by 0
  9. 2026InTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Searchcited by 0