# Depth-Aware Initialization for Stable and Efficient Neural Network Training

arXiv: https://arxiv.org/abs/2509.05018 · HTML: https://mlsanity.com/paper/2509.05018/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 7 references that are in the map.
- First posted on arXiv 5 Sep 2025.

## Builds on (most cited first, 4 of 4)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [All you need is a good init](https://mlsanity.com/paper/1511.06422/) (2015)
- [Random Walk Initialization for Training Very Deep Feedforward Networks](https://mlsanity.com/paper/1412.6558/) (2014)

## Abstract (arXiv)

In past few years, various initialization schemes have been proposed. These schemes are glorot initialization, He initialization, initialization using orthogonal matrix, random walk method for initialization. Some of these methods stress on keeping unit variance of activation and gradient propagation through the network layer. Few of these methods are independent of the depth information while some methods has considered the total network depth for better initialization. In this paper, comprehensive study has been done where depth information of each layer as well as total network is incorporated for better initialization scheme. It has also been studied that for deeper networks theoretical assumption of unit variance throughout the network does not perform well. It requires the need to increase the variance of the network from first layer activation to last layer activation. We proposed a novel way to increase the variance of the network in flexible manner, which incorporates the information of each layer depth. Experiments shows that proposed method performs better than the existing initialization scheme.