mlsanityData to 14 Oct 2026

Bag of Tricks for Neural Architecture Search

arXiv 2107.03719 ↗First posted 8 Jul 2021cs.LG cs.AI stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

While neural architecture search methods have been successful in previous years and led to new state-of-the-art performance on various problems, they have also been criticized for being unstable, being highly sensitive with respect to their hyperparameters, and often not performing better than random search. To shed some light on this issue, we discuss some practical considerations that help improve the stability, efficiency and overall performance.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 8 Jul 2021.

Builds on

45 of 59 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Layer Normalizationcited by 5,669
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2019Searching for MobileNetV3cited by 1,994
  8. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  9. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  10. 2018Group Normalizationcited by 1,524
  11. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  12. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
33 more
  1. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  2. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 414
  3. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 344
  4. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  5. 2019Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentationcited by 260
  6. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 254
  7. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 178
  8. 2019Evaluating the Search Phase of Neural Architecture Searchcited by 120
  9. 2019Exploring Randomly Wired Neural Networks for Image Recognitioncited by 103
  10. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  11. 2018Searching for Efficient Multi-Scale Architectures for Dense Image Predictioncited by 96
  12. 2017Shake-Shake regularizationcited by 87
  13. 2021HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmarkcited by 84
  14. 2019DetNAS: Backbone Search for Object Detectioncited by 81
  15. 2020DARTS-: Robustly Stepping out of Performance Collapse Without Indicatorscited by 81
  16. 2019Meta-Learning of Neural Architectures for Few-Shot Learningcited by 73
  17. 2019A Survey on Neural Architecture Searchcited by 71
  18. 2019NAS evaluation is frustratingly hardcited by 70
  19. 2020Can weight sharing outperform random architecture search? An investigation with TuNAScited by 68
  20. 2017MegDet: A Large Mini-Batch Object Detectorcited by 66
  21. 2020NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Searchcited by 64
  22. 2020EcoNAS: Finding Proxies for Economical Neural Architecture Searchcited by 61
  23. 2019Best Practices for Scientific Research on Neural Architecture Searchcited by 54
  24. 2019NAS-FCOS: Fast Neural Architecture Search for Object Detectioncited by 53
  25. 2019SpineNet: Learning Scale-Permuted Backbone for Recognition and Localizationcited by 50
  26. 2020Few-shot Neural Architecture Searchcited by 50
  27. 2018Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cellscited by 39
  28. 2020Hit-Detector: Hierarchical Trinity Architecture Search for Object Detectioncited by 32
  29. 2019SqueezeNAS: Fast neural architecture search for faster semantic segmentationcited by 31
  30. 2020Neural Architecture Generator Optimizationcited by 30
  31. 2019AutoDispNet: Improving Disparity Estimation With AutoMLcited by 29
  32. 2019MnasFPN: Learning Latency-aware Pyramid Architecture for Object Detection on Mobile Devicescited by 18
  33. 2019Efficient Neural Architecture Transformation Searchin Channel-Level for Object Detectioncited by 16

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 1; 2023: 1.

4 papers.

  1. 2022Neural Architecture Search for Dense Prediction Tasks in Computer Visioncited by 12
  2. 2023Are Neural Architecture Search Benchmarks Well Designed? A Deeper Look Into Operation Importancecited by 3
  3. 2021Training BatchNorm Only in Neural Architecture Search and Beyondcited by 2
  4. 2021NeuralArTS: Structuring Neural Architecture Search with Type Theorycited by 0