# BN-NAS: Neural Architecture Search with Batch Normalization

arXiv: https://arxiv.org/abs/2108.07375 · HTML: https://mlsanity.com/paper/2108.07375/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 37 references that are in the map.
- First posted on arXiv 16 Aug 2021.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation](https://mlsanity.com/paper/1904.12760/) (2019)
- [Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks](https://mlsanity.com/paper/1909.11957/) (2019)
- [FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions](https://mlsanity.com/paper/2004.05565/) (2020)
- [FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search](https://mlsanity.com/paper/1907.01845/) (2019)
- [Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs](https://mlsanity.com/paper/2003.00152/) (2020)
- [Searching for Efficient Multi-Scale Architectures for Dense Image Prediction](https://mlsanity.com/paper/1809.04184/) (2018)
- [Neural Optimizer Search with Reinforcement Learning](https://mlsanity.com/paper/1709.07417/) (2017)
- [DetNAS: Backbone Search for Object Detection](https://mlsanity.com/paper/1903.10979/) (2019)
- [CARS: Continuous Evolution for Efficient Neural Architecture Search](https://mlsanity.com/paper/1909.04977/) (2019)
- [Operation-Aware Soft Channel Pruning using Differentiable Masks](https://mlsanity.com/paper/2007.03938/) (2020)
- [GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet](https://mlsanity.com/paper/2003.11236/) (2020)
- [EcoNAS: Finding Proxies for Economical Neural Architecture Search](https://mlsanity.com/paper/2001.01233/) (2020)
- [GLiT: Neural Architecture Search for Global and Local Image Transformer](https://mlsanity.com/paper/2107.02960/) (2021)
- [Angle-based Search Space Shrinking for Neural Architecture Search](https://mlsanity.com/paper/2004.13431/) (2020)
- [Improving One-shot NAS by Suppressing the Posterior Fading](https://mlsanity.com/paper/1910.02543/) (2019)

## Built on by (most cited first, 21 of 21)

- [Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space](https://mlsanity.com/paper/2201.00814/) (2022)
- [ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients](https://mlsanity.com/paper/2301.11300/) (2023)
- [Mixed-Precision Neural Network Quantization via Learned Layer-wise Importance](https://mlsanity.com/paper/2203.08368/) (2022)
- [Neural Architecture Search for Spiking Neural Networks](https://mlsanity.com/paper/2201.10355/) (2022)
- [AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search](https://mlsanity.com/paper/2403.19232/) (2024)
- [Normalization Layers Are All That Sharpness-Aware Minimization Needs](https://mlsanity.com/paper/2306.04226/) (2023)
- [Analyzing and Mitigating Interference in Neural Architecture Search](https://mlsanity.com/paper/2108.12821/) (2021)
- [LightCTS: A Lightweight Framework for Correlated Time Series Forecasting](https://mlsanity.com/paper/2302.11974/) (2023)
- [FIT: A Metric for Model Sensitivity](https://mlsanity.com/paper/2210.08502/) (2022)
- [Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing](https://mlsanity.com/paper/2210.04153/) (2022)
- [Stimulative Training++: Go Beyond The Performance Limits of Residual Networks](https://mlsanity.com/paper/2305.02507/) (2023)
- [From Tiny Machine Learning to Tiny Deep Learning: A Survey](https://mlsanity.com/paper/2506.18927/) (2025)
- [Differentiable Architecture Search with Random Features](https://mlsanity.com/paper/2208.08835/) (2022)
- [When NAS Meets Trees: An Efficient Algorithm for Neural Architecture Search](https://mlsanity.com/paper/2204.04918/) (2022)
- [Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights](https://mlsanity.com/paper/2502.04975/) (2025)
- [Training BatchNorm Only in Neural Architecture Search and Beyond](https://mlsanity.com/paper/2112.00265/) (2021)
- [EM-DARTS: Hierarchical Differentiable Architecture Search for Eye Movement Recognition](https://mlsanity.com/paper/2409.14432/) (2024)
- [Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvature](https://mlsanity.com/paper/2508.12977/) (2025)
- [Improving Differentiable Architecture Search with a Generative Model](https://mlsanity.com/paper/2112.00171/) (2021)
- [Dynamical Isometry based Rigorous Fair Neural Architecture Search](https://mlsanity.com/paper/2307.02263/) (2023)
- [InTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Search](https://mlsanity.com/paper/2606.18676/) (2026)

## Abstract (arXiv)

We present BN-NAS, neural architecture search with Batch Normalization (BN-NAS), to accelerate neural architecture search (NAS). BN-NAS can significantly reduce the time required by model training and evaluation in NAS. Specifically, for fast evaluation, we propose a BN-based indicator for predicting subnet performance at a very early training stage. The BN-based indicator further facilitates us to improve the training efficiency by only training the BN parameters during the supernet training. This is based on our observation that training the whole supernet is not necessary while training only BN parameters accelerates network convergence for network architecture search. Extensive experiments show that our method can significantly shorten the time of training supernet by more than 10 times and shorten the time of evaluating subnets by more than 600,000 times without losing accuracy.