mlsanityData to 7 Oct 2026

Splitting Steepest Descent for Growing Neural Architectures

arXiv 1910.02366 ↗First posted 6 Oct 2019cs.LG cs.NE stat.MLcited by 52

Not on the map. Why?

Abstract (arXiv)

We develop a progressive training approach for neural networks which adaptively grows the network structure by splitting existing neurons to multiple off-springs. By leveraging a functional steepest descent idea, we derive a simple criterion for deciding the best subset of neurons to split and a splitting gradient for optimally updating the off-springs. Theoretically, our splitting strategy is a second-order functional steepest descent for escaping saddle points in an ∞-Wasserstein metric space, on which the standard parametric gradient descent is a first-order steepest descent. Our method provides a new computationally efficient approach for optimizing neural network structures, especially for learning lightweight neural architectures in resource-constrained settings.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 9; 2023: 9; 2024: 10; 2025: 8; 2026: 8.

52 papers.

Sort:

By area:

  1. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  2. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  3. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  4. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
  5. 2022Staged Training for Transformer Language Modelscited by 49
  6. 2021Firefly Neural Architecture Descent: a General Approach for Growing Neural Networkscited by 48
  7. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  8. 2022Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEscited by 25
  9. 2021The Elastic Lottery Ticket Hypothesiscited by 23
  10. 2021On Anytime Learning at Macroscalecited by 20
  11. 2021PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splittingcited by 20
  12. 2022When, where, and how to add new neurons to ANNscited by 18
38 more
  1. 2022Auto-scaling Vision Transformers without Trainingcited by 15
  2. 2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
  3. 2022Streamable Neural Fieldscited by 13
  4. 2023Self-Expanding Neural Networkscited by 13
  5. 2025Steepest Descent Density Control for Compact 3D Gaussian Splattingcited by 13
  6. 2024Preparing Lessons for Progressive Training on Language Modelscited by 10
  7. 2024Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimallycited by 9
  8. 2023Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptationcited by 8
  9. 2023MixtureGrowth: Growing Neural Networks by Recombining Learned Parameterscited by 7
  10. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  11. 2024DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusioncited by 6
  12. 2021Doping: A technique for efficient compression of LSTM models using sparse structured additive matricescited by 5
  13. 2024Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Netscited by 5
  14. 2023Split-Ensemble: Efficient OOD-aware Ensemble via Task and Model Splittingcited by 4
  15. 2025Adaptive Width Neural Networkscited by 4
  16. 2026SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learningcited by 4
  17. 2024When To Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networkscited by 3
  18. 2025Energy-Aware Deep Learning on Resource-Constrained Hardwarecited by 3
  19. 2025Flat Channels to Infinity in Neural Loss Landscapescited by 3
  20. 2022An Adaptive and Stability-Promoting Layerwise Training Approach for Sparse Deep Neural Network Architecturecited by 2
  21. 2025Topological derivative approach for deep neural network architecture adaptationcited by 2
  22. 2026Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generationcited by 2
  23. 2021Differentiable Neural Architecture Search with Morphism-based Transformable Backbone Architecturescited by 1
  24. 2023SensLI: Sensitivity-Based Layer Insertion for Neural Networkscited by 1
  25. 2024Learning Morphisms with Gauss-Newton Approximation for Growing Networkscited by 1
  26. 2025Variational Kolmogorov-Arnold Networkcited by 1
  27. 2025Saddle Hierarchy in Dense Associative Memorycited by 1
  28. 2025Shared-Weights Extender and Gradient Voting for Neural Network Expansioncited by 1
  29. 2026Data Reconstruction: Identifiability and Optimization with Sample Splittingcited by 1
  30. 2021Growing Representation Learningcited by 0
  31. 2022Growing Neural Network with Shared Parametercited by 0
  32. 2023Energy Concerns with HPC Systems and Applicationscited by 0
  33. 2024Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architecturescited by 0
  34. 2024Growing Deep Neural Network Considering with Similarity between Neuronscited by 0
  35. 2024Growing Efficient Accurate and Robust Neural Networks on the Edgecited by 0
  36. 2026Growing Networks with Autonomous Pruningcited by 0
  37. 2026On the Stability of Growth in Structural Plasticity: Forward-Active yet Backward-Starvedcited by 0
  38. 2026An optimal control approach for neural network architecture adaptation with a posteriori error estimationcited by 0

50 most cited of 52 All 52 (JSON)