mlsanityData to 7 Oct 2026

Neural Architecture Search via Bregman Iterations

arXiv 2106.02479 ↗First posted 4 Jun 2021cs.LG cs.NE math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

We propose a novel strategy for Neural Architecture Search (NAS) based on Bregman iterations. Starting from a sparse neural network our gradient-based one-shot algorithm gradually adds relevant parameters in an inverse scale space manner. This allows the network to choose the best architecture in the search space which makes it well-designed for a given task, e.g., by adding neurons or skip connections. We demonstrate that using our approach one can unveil, for instance, residual autoencoders for denoising, deblurring, and classification tasks. Code is available at https://github.com/TimRoith/BregmanLearning.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 4 Jun 2021.

Builds on

33 of 56 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2018Neural Ordinary Differential Equationscited by 3,733
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  10. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  11. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  12. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
21 more
  1. 2016An overview of gradient descent optimization algorithmscited by 1,021
  2. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  3. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  4. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  5. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  6. 2018Efficient Multi-objective Neural Architecture Search via Lamarckian Evolutioncited by 151
  7. 2017Evolving Deep Neural Networkscited by 93
  8. 2016Group Sparse Regularization for Deep Neural Networkscited by 89
  9. 2019BayesNAS: A Bayesian Approach for Neural Architecture Searchcited by 67
  10. 2017NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigmcited by 60
  11. 2019Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardwarecited by 48
  12. 2021Momentum Residual Neural Networkscited by 36
  13. 2019Deep learning as optimal control problems: models and numerical methodscited by 33
  14. 2018You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimizationcited by 23
  15. 2021A Bregman Learning Framework for Sparse Neural Networkscited by 15
  16. 2019The Role of Memory in Stochastic Optimizationcited by 10
  17. 2020Neural Architecture Search as Sparse Supernetcited by 10
  18. 2017Choose your path wisely: gradient descent in a Bregman distance frameworkcited by 8
  19. 2019Exploring Structural Sparsity of Deep Networks via Inverse Scale Spacescited by 7
  20. 2020Semi-discrete optimization through semi-discrete optimal transport: a framework for neural architecture searchcited by 2
  21. 2020Traditional and accelerated gradient descent for neural architecture searchcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 0; 2025: 1; 2026: 1.

5 papers.

  1. 2022Duality for Neural Networks through Reproducing Kernel Banach Spacescited by 15
  2. 2023From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networkscited by 3
  3. 2026Sparse Training of Neural Networks based on Multilevel Mirror Descentcited by 3
  4. 2023Learning a Sparse Representation of Barron Functions with the Inverse Scale Space Flowcited by 1
  5. 2025OCTANE -- Optimal Control for Tensor-based Autoencoder Network Emergence: Explicit Casecited by 0