mlsanityData to 7 Oct 2026

Generalization Properties of NAS under Activation and Skip Connection Search

arXiv 2209.07238 ↗First posted 15 Sep 2022cs.LG cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Neural Architecture Search (NAS) has fostered the automatic discovery of state-of-the-art neural architectures. Despite the progress achieved with NAS, so far there is little attention to theoretical guarantees on NAS. In this work, we study the generalization properties of NAS under a unifying framework enabling (deep) layer skip connection search and activation function search. To this end, we derive the lower (and upper) bounds of the minimum eigenvalue of the Neural Tangent Kernel (NTK) under the (in)finite-width regime using a certain search space including mixed activation functions, fully connected, and residual neural networks. We use the minimum eigenvalue to establish generalization error bounds of NAS in the stochastic gradient descent training. Importantly, we theoretically and experimentally show how the derived results can guide NAS to select the top-performing architectures, even in the case without training, leading to a train-free algorithm based on our theory. Accordingly, our numerical validation shed light on the design of computationally efficient methods for NAS. Our analysis is non-trivial due to the coupling of various architectures and activation functions under the unifying framework and has its own interest in providing the lower bound of the minimum eigenvalue of NTK in deep learning theory.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 52 of its 87 references that are in the map. First posted on arXiv 15 Sep 2022.

Builds on

52 of 87 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  7. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  10. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  11. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
38 more
  1. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  2. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  3. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  4. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  5. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  6. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  7. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  8. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  9. 2019Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentationcited by 258
  10. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  11. 2020A Comprehensive Survey of Neural Architecture Search: Challenges and Solutionscited by 253
  12. 2020Neural Architecture Search without Trainingcited by 241
  13. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  14. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  15. 2017Hierarchical Representations for Efficient Architecture Searchcited by 188
  16. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  17. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  18. 2020NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Sizecited by 98
  19. 2021Learning with invariances in random features and kernel modelscited by 89
  20. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  21. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  22. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  23. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  24. 2019DetNAS: Backbone Search for Object Detectioncited by 80
  25. 2019On Network Design Spaces for Visual Recognitioncited by 68
  26. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  27. 2022β-DARTS: Beta-Decay Regularization for Differentiable Architecture Searchcited by 63
  28. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  29. 2019Understanding Architectures Learnt by Cell-based Neural Architecture Searchcited by 48
  30. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  31. 2019Customizable Architecture Search for Semantic Segmentationcited by 41
  32. 2021KNAS: Green Neural Architecture Searchcited by 33
  33. 2021NASI: Label- and Data-agnostic Neural Architecture Search at Initializationcited by 28
  34. 2020Theory-Inspired Path-Regularized Differential Network Architecture Searchcited by 25
  35. 2021The Three Stages of Learning Dynamics in High-Dimensional Kernel Methodscited by 21
  36. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 20
  37. 2020Kernel-Based Smoothness Analysis of Residual Networkscited by 19
  38. 2020AutoSpeech: Neural Architecture Search for Speaker Recognitioncited by 17

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 4; 2026: 1.

14 papers.

Sort:

  1. 2023On the Convergence of Encoder-only Shallow Transformerscited by 13
  2. 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10
  3. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  4. 2024Robust NAS under adversarial training: benchmark, theory, and beyondcited by 6
  5. 2023LayerNAS: Neural Architecture Search in Polynomial Complexitycited by 3
  6. 2024On the Generalization Capability of Temporal Graph Learning Algorithms: Theoretical Insights and a Simpler Methodcited by 3
  7. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  8. 2024The effect of Leaky ReLUs on the training and generalization of overparameterized networkscited by 2
  9. 2025DiffoRA: Enabling Parameter-Efficient Fine-Tuning via Differential Module Selectioncited by 1
  10. 2025Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvaturecited by 1
  11. 2023Neural Tangent Kernels Motivate Graph Neural Networks with Cross-Covariance Graphscited by 0
  12. 2024ActNAS : Generating Efficient YOLO Models using Activation NAScited by 0
2 more
  1. 2025Is Supervised Learning Really That Different from Unsupervised?cited by 0
  2. 2026Zero-shot Quantum Neural Architecture Searchcited by 0