# When, where, and how to add new neurons to ANNs

arXiv: https://arxiv.org/abs/2202.08539 · HTML: https://mlsanity.com/paper/2202.08539/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 17 Feb 2022.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [Neural Architecture Search without Training](https://mlsanity.com/paper/2006.04647/) (2020)
- [FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions](https://mlsanity.com/paper/2004.05565/) (2020)
- [Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2010.14498/) (2020)
- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [Network Morphism](https://mlsanity.com/paper/1603.01670/) (2016)
- [NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm](https://mlsanity.com/paper/1711.02017/) (2017)
- [Splitting Steepest Descent for Growing Neural Architectures](https://mlsanity.com/paper/1910.02366/) (2019)
- [Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks](https://mlsanity.com/paper/2102.08574/) (2021)
- [Batch Normalization Orthogonalizes Representations in Deep Random Networks](https://mlsanity.com/paper/2106.03970/) (2021)
- [AutoGrow: Automatic Layer Growing in Deep Convolutional Networks](https://mlsanity.com/paper/1906.02909/) (2019)
- [Efficient Network Construction through Structural Plasticity](https://mlsanity.com/paper/1905.11530/) (2019)

## Built on by (most cited first, 18 of 18)

- [Towards Self-Assembling Artificial Neural Networks through Neural Developmental Programs](https://mlsanity.com/paper/2307.08197/) (2023)
- [Self-Expanding Neural Networks](https://mlsanity.com/paper/2307.04526/) (2023)
- [Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimally](https://mlsanity.com/paper/2405.19816/) (2024)
- [MixtureGrowth: Growing Neural Networks by Recombining Learned Parameters](https://mlsanity.com/paper/2311.04251/) (2023)
- [Augmentative Topology Agents For Open-Ended Learning](https://mlsanity.com/paper/2210.11442/) (2022)
- [When To Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks](https://mlsanity.com/paper/2401.03104/) (2024)
- [Growing Artificial Neural Networks for Control: the Role of Neuronal Diversity](https://mlsanity.com/paper/2405.08510/) (2024)
- [Evolved Developmental Artificial Neural Networks for Multitasking with Advanced Activity Dependence](https://mlsanity.com/paper/2407.10359/) (2024)
- [Growth strategies for arbitrary DAG neural architectures](https://mlsanity.com/paper/2501.12690/) (2025)
- [Topological derivative approach for deep neural network architecture adaptation](https://mlsanity.com/paper/2502.06885/) (2025)
- [Foundations of a Developmental Design Paradigm for Integrated Continual Learning, Deliberative Behavior, and Comprehensibility](https://mlsanity.com/paper/2502.13935/) (2025)
- [SensLI: Sensitivity-Based Layer Insertion for Neural Networks](https://mlsanity.com/paper/2311.15995/) (2023)
- [Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?](https://mlsanity.com/paper/2408.07588/) (2024)
- [Evolutionary Developmental Biology Can Serve as the Conceptual Foundation for a New Design Paradigm in Artificial Intelligence](https://mlsanity.com/paper/2506.12891/) (2025)
- [Enhancing Efficiency and Performance in Deepfake Audio Detection through Neuron-level Dropin & Neuroplasticity Mechanisms](https://mlsanity.com/paper/2603.24343/) (2026)
- [NORACL: Neurogenesis for Oracle-free Resource-Adaptive Continual Learning](https://mlsanity.com/paper/2604.27031/) (2026)
- [GRAPE: Guided Parameter-Space Evolution for Compact Adversarial Robustness](https://mlsanity.com/paper/2606.14865/) (2026)
- [CAGE-NAS: Certified Functional Descent for Efficient Model Growth](https://mlsanity.com/paper/2610.01173/) (2026)

## Abstract (arXiv)

Neurogenesis in ANNs is an understudied and difficult problem, even compared to other forms of structural learning like pruning. By decomposing it into triggers and initializations, we introduce a framework for studying the various facets of neurogenesis: when, where, and how to add neurons during the learning process. We present the Neural Orthogonality (NORTH*) suite of neurogenesis strategies, combining layer-wise triggers and initializations based on the orthogonality of activations or weights to dynamically grow performant networks that converge to an efficient size. We evaluate our contributions against other recent neurogenesis works across a variety of supervised learning tasks.