# Random Walk Initialization for Training Very Deep Feedforward Networks

arXiv: https://arxiv.org/abs/1412.6558 · HTML: https://mlsanity.com/paper/1412.6558/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2014.

## Built on by (most cited first, 13 of 13)

- [On Efficient Training of Large-Scale Deep Learning Models: A Literature Review](https://mlsanity.com/paper/2304.03589/) (2023)
- [High-performance deep spiking neural networks with 0.3 spikes per neuron](https://mlsanity.com/paper/2306.08744/) (2023)
- [DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs](https://mlsanity.com/paper/2112.04036/) (2021)
- [DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs](https://mlsanity.com/paper/2205.01938/) (2022)
- [Improving Deep Neural Network Random Initialization Through Neuronal Rewiring](https://mlsanity.com/paper/2207.08148/) (2022)
- [On the Initialization of Graph Neural Networks](https://mlsanity.com/paper/2312.02622/) (2023)
- [Data-driven Weight Initialization with Sylvester Solvers](https://mlsanity.com/paper/2105.10335/) (2021)
- [A Weight Initialization Based on the Linear Product Structure for Neural Networks](https://mlsanity.com/paper/2109.00125/) (2021)
- [DREAM: Debugging and Repairing AutoML Pipelines](https://mlsanity.com/paper/2401.00379/) (2023)
- [An Effective Weight Initialization Method for Deep Learning: Application to Satellite Image Classification](https://mlsanity.com/paper/2406.00348/) (2024)
- [Learning Enhancement of CNNs via Separation Index Maximizing at the First Convolutional Layer](https://mlsanity.com/paper/2201.05217/) (2022)
- [SPGD: Steepest Perturbed Gradient Descent Optimization](https://mlsanity.com/paper/2411.04946/) (2024)
- [Depth-Aware Initialization for Stable and Efficient Neural Network Training](https://mlsanity.com/paper/2509.05018/) (2025)

## Abstract (arXiv)

Training very deep networks is an important open problem in machine learning. One of many difficulties is that the norm of the back-propagated error gradient can grow or decay exponentially. Here we show that training very deep feed-forward networks (FFNs) is not as difficult as previously thought. Unlike when back-propagation is applied to a recurrent network, application to an FFN amounts to multiplying the error gradient by a different random matrix at each layer. We show that the successive application of correctly scaled random matrices to an initial vector results in a random walk of the log of the norm of the resulting vectors, and we compute the scaling that makes this walk unbiased. The variance of the random walk grows only linearly with network depth and is inversely proportional to the size of each layer. Practically, this implies a gradient whose log-norm scales with the square root of the network depth and shows that the vanishing gradient problem can be mitigated by increasing the width of the layers. Mathematical analyses and experimental results using stochastic gradient descent to optimize tasks related to the MNIST and TIMIT datasets are provided to support these claims. Equations for the optimal matrix scaling are provided for the linear and ReLU cases.