mlsanityData to 7 Oct 2026

Self-Expanding Neural Networks

arXiv 2307.04526 ↗First posted 10 Jul 2023cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

The results of training a neural network are heavily dependent on the architecture chosen; and even a modification of only its size, however small, typically involves restarting the training process. In contrast to this, we begin training with a small architecture, only increase its capacity as necessary for the problem, and avoid interfering with previous optimization while doing so. We thereby introduce a natural gradient based approach which intuitively expands both the width and depth of a neural network when this is likely to substantially reduce the hypothetical converged training loss. We prove an upper bound on the ``rate'' at which neurons are added, and a computationally cheap lower bound on the expansion score. We illustrate the benefits of such Self-Expanding Neural Networks with full connectivity and convolutions in both classification and regression problems, including those where the appropriate architecture size is substantially uncertain a priori.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 18 of its 39 references that are in the map. First posted on arXiv 10 Jul 2023.

Builds on

18 of 39 references.

References · 18

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2014Going Deeper with Convolutionscited by 4,231
  3. 2016Progressive Neural Networkscited by 1,332
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  5. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  6. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  7. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  8. 2014New insights and perspectives on the natural gradient methodcited by 435
  9. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  10. 2020A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learningcited by 86
  11. 2019Pad'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networkscited by 65
  12. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
6 more
  1. 2019Splitting Steepest Descent for Growing Neural Architecturescited by 52
  2. 2021Firefly Neural Architecture Descent: a General Approach for Growing Neural Networkscited by 48
  3. 2016Neurogenesis Deep Learningcited by 32
  4. 2021Knowledge-Adaptation Priorscited by 20
  5. 2022When, where, and how to add new neurons to ANNscited by 18
  6. 2018Deep Active Learning with a Neural Architecture Searchcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 5; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Continual Learning and Catastrophic Forgettingcited by 73
  2. 2023Adaptive Message Passing: A General Framework to Mitigate Oversmoothing, Oversquashing, and Underreachingcited by 23
  3. 2023Navigating Scaling Laws: Compute Optimality in Adaptive Model Trainingcited by 4
  4. 2025Adaptive Width Neural Networkscited by 4
  5. 2025Foundations of a Developmental Design Paradigm for Integrated Continual Learning, Deliberative Behavior, and Comprehensibilitycited by 2
  6. 2026Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generationcited by 2
  7. 2023SensLI: Sensitivity-Based Layer Insertion for Neural Networkscited by 1
  8. 2024Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?cited by 1
  9. 2025Growing with Experience: Growing Neural Networks in Deep Reinforcement Learningcited by 1
  10. 2025Variational Kolmogorov-Arnold Networkcited by 1
  11. 2025Shared-Weights Extender and Gradient Voting for Neural Network Expansioncited by 1
  12. 2024Self Expanding Convolutional Neural Networkscited by 0
1 more
  1. 2026Fast and Slow Variational Continual Learningcited by 0