# Gradual DropIn of Layers to Train Very Deep Neural Networks

arXiv: https://arxiv.org/abs/1511.06951 · HTML: https://mlsanity.com/paper/1511.06951/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2015.

## Built on by (most cited first, 8 of 8)

- [Ensemble deep learning: A review](https://mlsanity.com/paper/2104.02395/) (2021)
- [Automated Progressive Learning for Efficient Training of Vision Transformers](https://mlsanity.com/paper/2203.14509/) (2022)
- [On Efficient Training of Large-Scale Deep Learning Models: A Literature Review](https://mlsanity.com/paper/2304.03589/) (2023)
- [Progressive Transformation Learning for Leveraging Virtual Images in Training](https://mlsanity.com/paper/2211.01778/) (2022)
- [General Cyclical Training of Neural Networks](https://mlsanity.com/paper/2202.08835/) (2022)
- [Efficient Training of Large Vision Models via Advanced Automated Progressive Learning](https://mlsanity.com/paper/2410.00350/) (2024)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Efficient Training for Human Video Generation with Entropy-Guided Prioritized Progressive Learning](https://mlsanity.com/paper/2511.21136/) (2025)

## Abstract (arXiv)

We introduce the concept of dynamically growing a neural network during training. In particular, an untrainable deep network starts as a trainable shallow network and newly added layers are slowly, organically added during training, thereby increasing the network's depth. This is accomplished by a new layer, which we call DropIn. The DropIn layer starts by passing the output from a previous layer (effectively skipping over the newly added layers), then increasingly including units from the new layers for both feedforward and backpropagation. We show that deep networks, which are untrainable with conventional methods, will converge with DropIn layers interspersed in the architecture. In addition, we demonstrate that DropIn provides regularization during training in an analogous way as dropout. Experiments are described with the MNIST dataset and various expanded LeNet architectures, CIFAR-10 dataset with its architecture expanded from 3 to 11 layers, and on the ImageNet dataset with the AlexNet architecture expanded to 13 layers and the VGG 16-layer architecture.