mlsanityData to 7 Oct 2026

Geometry-Aware Gradient Algorithms for Neural Architecture Search

arXiv 2004.07802 ↗First posted 16 Apr 2020cs.LG cs.CV cs.NEcited by 40

Not on the map. Why?

Abstract (arXiv)

Recent state-of-the-art methods for neural architecture search (NAS) exploit gradient-based optimization by relaxing the problem into continuous optimization over architectures and shared-weights, a noisy process that remains poorly understood. We argue for the study of single-level empirical risk minimization to understand NAS with weight-sharing, reducing the design of NAS methods to devising optimizers and regularizers that can quickly obtain high-quality solutions to this problem. Invoking the theory of mirror descent, we present a geometry-aware framework that exploits the underlying structure of this optimization to return sparse architectural parameters, leading to simple yet novel algorithms that enjoy fast convergence guarantees and achieve state-of-the-art accuracy on the latest NAS benchmarks in computer vision. Notably, we exceed the best published results for both CIFAR and ImageNet on both the DARTS search space and NAS-Bench201; on the latter we achieve near-oracle-optimal performance on CIFAR-10 and CIFAR-100. Together, our theory and experiments demonstrate a principled way to co-design optimizers and continuous relaxations of discrete NAS search spaces.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 9; 2023: 7; 2024: 4; 2025: 3; 2026: 2.

40 papers.

Sort:

  1. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  2. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 88
  3. 2021Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharingcited by 58
  4. 2021NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Taskscited by 32
  5. 2022Generalizing Few-Shot NAS with Gradient Matchingcited by 25
  6. 2021NAS-Bench-x11 and the Power of Learning Curvescited by 20
  7. 2021Rethinking Neural Operations for Diverse Taskscited by 18
  8. 2022On Redundancy and Diversity in Cell-based Neural Architecture Searchcited by 18
  9. 2022Efficient Architecture Search for Diverse Taskscited by 18
  10. 2021Sphynx: ReLU-Efficient Network Design for Private Inferencecited by 15
  11. 2021Generalization Guarantees for Neural Architecture Search with Train-Validation Splitcited by 14
  12. 2022AANG: Automating Auxiliary Learningcited by 12
28 more
  1. 2022A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithmscited by 8
  2. 2021DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecturecited by 6
  3. 2021L2NAS: Learning to Optimize Neural Architectures via Continuous-Action Reinforcement Learningcited by 6
  4. 2021Approximate Neural Architecture Search via Operation Distribution Learningcited by 6
  5. 2022Speeding up NAS with Adaptive Subset Selectioncited by 6
  6. 2025Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Modelscited by 5
  7. 2021Differentiable Architecture Search for Reinforcement Learningcited by 4
  8. 2021Profiling Neural Blocks and Design Spaces for Mobile Neural Architecture Searchcited by 4
  9. 2023Weight-Entanglement Meets Gradient-Based Neural Architecture Searchcited by 4
  10. 2022Federated Hypergradient Descentcited by 2
  11. 2023Rethink DARTS Search Space and Renovate a New Benchmarkcited by 2
  12. 2023Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularizationcited by 2
  13. 2024Pretrained Hybrids with MAD Skillscited by 2
  14. 2026Geometric Neural Operators via Lie Group-Constrained Latent Dynamicscited by 2
  15. 2021Self-Constructing Neural Networks Through Random Mutationcited by 1
  16. 2022A Deep Neural Networks ensemble workflow from hyperparameter search to inference leveraging GPU clusterscited by 1
  17. 2023Small Temperature is All You Need for Differentiable Architecture Searchcited by 1
  18. 2023Balanced Mixture of SuperNets for Learning the CNN Pooling Architecturecited by 1
  19. 2023CycleGANAS: Differentiable Neural Architecture Search for CycleGANcited by 1
  20. 2024MoRe Fine-Tuning with 10x Fewer Parameterscited by 1
  21. 2021Neural Architecture Search in operational context: a remote sensing case-studycited by 0
  22. 2021Delve into the Performance Degradation of Differentiable Architecture Searchcited by 0
  23. 2022SUTD-PRCM Dataset and Neural Architecture Search Approach for Complex Metasurface Designcited by 0
  24. 2024ECToNAS: Evolutionary Cross-Topology Neural Architecture Searchcited by 0
  25. 2024CSCO: Connectivity Search of Convolutional Operatorscited by 0
  26. 2025confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methodscited by 0
  27. 2025Evolutionary Neural Architecture Search with Dual Contrastive Learningcited by 0
  28. 2026How Much Does Message Passing Matter? A Drop-In Study of GNN Layers for Neural Network Graph Regressioncited by 0