mlsanityData to 7 Oct 2026

Learning Morphisms with Gauss-Newton Approximation for Growing Networks

arXiv 2411.05855 ↗First posted 7 Nov 2024cs.LG cs.CV cs.NEcited by 1

Not on the map. Why?

Abstract (arXiv)

A popular method for Neural Architecture Search (NAS) is based on growing networks via small local changes to the network's architecture called network morphisms. These methods start with a small seed network and progressively grow the network by adding new neurons in an automated way. However, it remains a challenge to efficiently determine which parts of the network are best to grow. Here we propose a NAS method for growing a network by using a Gauss-Newton approximation of the loss function to efficiently learn and evaluate candidate network morphisms. We compare our method with state of the art NAS methods for CIFAR-10 and CIFAR-100 classification tasks, and conclude our method learns similar quality or better architectures at a smaller computational cost.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 39 of its 47 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

39 of 47 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  7. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  8. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2019Searching for MobileNetV3cited by 1,984
  12. 2018DARTS: Differentiable Architecture Searchcited by 1,782
27 more
  1. 2016Progressive Neural Networkscited by 1,332
  2. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  3. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  4. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  5. 2016Deep Networks with Stochastic Depthcited by 973
  6. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  7. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  9. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  10. 2018Rethinking the Value of Network Pruningcited by 595
  11. 2017Progressive Neural Architecture Searchcited by 483
  12. 2017Large-Scale Evolution of Image Classifierscited by 348
  13. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  14. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 200
  15. 2018Neural Architecture Optimizationcited by 164
  16. 2016Deep Pyramidal Residual Networkscited by 163
  17. 2018Efficient Multi-objective Neural Architecture Search via Lamarckian Evolutioncited by 151
  18. 2019AutoGAN: Neural Architecture Search for Generative Adversarial Networkscited by 94
  19. 2017MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networkscited by 92
  20. 2019Splitting Steepest Descent for Growing Neural Architecturescited by 52
  21. 2021Firefly Neural Architecture Descent: a General Approach for Growing Neural Networkscited by 48
  22. 2017Simple And Efficient Architecture Search for Convolutional Neural Networkscited by 40
  23. 2018Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cellscited by 39
  24. 2017DeepArchitect: Automatically Designing and Training Deep Architecturescited by 28
  25. 2019Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descentcited by 17
  26. 2018Transfer Learning with Neural AutoMLcited by 15
  27. 2020Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splittingcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Certified Inference and Training for Deep Equilibrium Networks: A Continuation Framework with Polynomial Complexity Guaranteescited by 0