# AutoGrow: Automatic Layer Growing in Deep Convolutional Networks

arXiv: https://arxiv.org/abs/1906.02909 · HTML: https://mlsanity.com/paper/1906.02909/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2019.

## Built on by (most cited first, 26 of 26)

- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks](https://mlsanity.com/paper/2102.08574/) (2021)
- [Automated Progressive Learning for Efficient Training of Vision Transformers](https://mlsanity.com/paper/2203.14509/) (2022)
- [Masked Structural Growth for 2x Faster Language Model Pre-training](https://mlsanity.com/paper/2305.02869/) (2023)
- [On Anytime Learning at Macroscale](https://mlsanity.com/paper/2106.09563/) (2021)
- [When, where, and how to add new neurons to ANNs](https://mlsanity.com/paper/2202.08539/) (2022)
- [GROWN: GRow Only When Necessary for Continual Learning](https://mlsanity.com/paper/2110.00908/) (2021)
- [DyRep: Bootstrapping Training with Dynamic Re-parameterization](https://mlsanity.com/paper/2203.12868/) (2022)
- [Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation](https://mlsanity.com/paper/2306.12700/) (2023)
- [MixtureGrowth: Growing Neural Networks by Recombining Learned Parameters](https://mlsanity.com/paper/2311.04251/) (2023)
- [Efficient Training of Large Vision Models via Advanced Automated Progressive Learning](https://mlsanity.com/paper/2410.00350/) (2024)
- [LESA: Learnable LLM Layer Scaling-Up](https://mlsanity.com/paper/2502.13794/) (2025)
- [Transformed CNNs: recasting pre-trained convolutional layers with self-attention](https://mlsanity.com/paper/2106.05795/) (2021)
- [When To Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks](https://mlsanity.com/paper/2401.03104/) (2024)
- [Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Search](https://mlsanity.com/paper/2307.15621/) (2023)
- [A General and Efficient Training for Transformer via Token Expansion](https://mlsanity.com/paper/2404.00672/) (2024)
- [Topological derivative approach for deep neural network architecture adaptation](https://mlsanity.com/paper/2502.06885/) (2025)
- [Compact CNN Structure Learning by Knowledge Distillation](https://mlsanity.com/paper/2104.09191/) (2021)
- [SensLI: Sensitivity-Based Layer Insertion for Neural Networks](https://mlsanity.com/paper/2311.15995/) (2023)
- [Overcoming Growth-Induced Forgetting in Task-Agnostic Continual Learning](https://mlsanity.com/paper/2408.10566/) (2024)
- [Growing with Experience: Growing Neural Networks in Deep Reinforcement Learning](https://mlsanity.com/paper/2506.11706/) (2025)
- [Differentiable Network Adaption with Elastic Search Space](https://mlsanity.com/paper/2103.16350/) (2021)
- [Compact Multi-level Sparse Neural Networks with Input Independent Dynamic Rerouting](https://mlsanity.com/paper/2112.10930/) (2021)
- [A Multi-Level Framework for Accelerating Training Transformer Models](https://mlsanity.com/paper/2404.07999/) (2024)
- [Growing Efficient Accurate and Robust Neural Networks on the Edge](https://mlsanity.com/paper/2410.07691/) (2024)
- [Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion](https://mlsanity.com/paper/2608.14664/) (2026)

## Abstract (arXiv)

Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts. We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth. We propose robust growing and stopping policies to generalize to different network architectures and datasets. Our experiments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet. For example, in terms of accuracy-computation trade-off, AutoGrow discovers a better depth combination in ResNets than human experts. Our AutoGrow is efficient. It discovers depth within similar time of training a single DNN. Our code is available at https://github.com/wenwei202/autogrow.