mlsanityData to 7 Oct 2026

When, where, and how to add new neurons to ANNs

arXiv 2202.08539 ↗First posted 17 Feb 2022cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Neurogenesis in ANNs is an understudied and difficult problem, even compared to other forms of structural learning like pruning. By decomposing it into triggers and initializations, we introduce a framework for studying the various facets of neurogenesis: when, where, and how to add neurons during the learning process. We present the Neural Orthogonality (NORTH*) suite of neurogenesis strategies, combining layer-wise triggers and initializations based on the orthogonality of activations or weights to dynamically grow performant networks that converge to an efficient size. We evaluate our contributions against other recent neurogenesis works across a variety of supervised learning tasks.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

19 of 32 references.

References · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Wide Residual Networkscited by 3,057
  4. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  5. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  6. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  7. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  8. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  9. 2020Neural Architecture Search without Trainingcited by 241
  10. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  11. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  12. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
7 more
  1. 2016Network Morphismcited by 63
  2. 2017NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigmcited by 60
  3. 2019Splitting Steepest Descent for Growing Neural Architecturescited by 52
  4. 2021Firefly Neural Architecture Descent: a General Approach for Growing Neural Networkscited by 48
  5. 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 28
  6. 2019AutoGrow: Automatic Layer Growing in Deep Convolutional Networkscited by 26
  7. 2019Efficient Network Construction through Structural Plasticitycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 5; 2025: 4; 2026: 4.

18 papers.

Sort:

  1. 2023Towards Self-Assembling Artificial Neural Networks through Neural Developmental Programscited by 17
  2. 2023Self-Expanding Neural Networkscited by 13
  3. 2024Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimallycited by 9
  4. 2023MixtureGrowth: Growing Neural Networks by Recombining Learned Parameterscited by 7
  5. 2022Augmentative Topology Agents For Open-Ended Learningcited by 3
  6. 2024When To Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networkscited by 3
  7. 2024Growing Artificial Neural Networks for Control: the Role of Neuronal Diversitycited by 2
  8. 2024Evolved Developmental Artificial Neural Networks for Multitasking with Advanced Activity Dependencecited by 2
  9. 2025Growth strategies for arbitrary DAG neural architecturescited by 2
  10. 2025Topological derivative approach for deep neural network architecture adaptationcited by 2
  11. 2025Foundations of a Developmental Design Paradigm for Integrated Continual Learning, Deliberative Behavior, and Comprehensibilitycited by 2
  12. 2023SensLI: Sensitivity-Based Layer Insertion for Neural Networkscited by 1
6 more
  1. 2024Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?cited by 1
  2. 2025Evolutionary Developmental Biology Can Serve as the Conceptual Foundation for a New Design Paradigm in Artificial Intelligencecited by 0
  3. 2026Enhancing Efficiency and Performance in Deepfake Audio Detection through Neuron-level Dropin & Neuroplasticity Mechanismscited by 0
  4. 2026NORACL: Neurogenesis for Oracle-free Resource-Adaptive Continual Learningcited by 0
  5. 2026GRAPE: Guided Parameter-Space Evolution for Compact Adversarial Robustnesscited by 0
  6. 2026CAGE-NAS: Certified Functional Descent for Efficient Model Growthcited by 0