# On weight initialization in deep neural networks

arXiv: https://arxiv.org/abs/1704.08863 · HTML: https://mlsanity.com/paper/1704.08863/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2017.

## Built on by (most cited first, 30 of 67)

- [Efficient Large Language Models: A Survey](https://mlsanity.com/paper/2312.03863/) (2023)
- [ControlNeXt: Powerful and Efficient Control for Image and Video Generation](https://mlsanity.com/paper/2408.06070/) (2024)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [On Efficient Training of Large-Scale Deep Learning Models: A Literature Review](https://mlsanity.com/paper/2304.03589/) (2023)
- [On Expressivity and Trainability of Quadratic Networks](https://mlsanity.com/paper/2110.06081/) (2021)
- [How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?](https://mlsanity.com/paper/2501.11253/) (2025)
- [Learning Invariant Weights in Neural Networks](https://mlsanity.com/paper/2202.12439/) (2022)
- [Disruptive Autoencoders: Leveraging Low-level features for 3D Medical Image Pre-training](https://mlsanity.com/paper/2307.16896/) (2023)
- [Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment](https://mlsanity.com/paper/2312.09257/) (2023)
- [DEBOSH: Deep Bayesian Shape Optimization](https://mlsanity.com/paper/2109.13337/) (2021)
- [Automatically Learning Hybrid Digital Twins of Dynamical Systems](https://mlsanity.com/paper/2410.23691/) (2024)
- [MoRe-Fi: Motion-robust and Fine-grained Respiration Monitoring via Deep-Learning UWB Radar](https://mlsanity.com/paper/2111.08195/) (2021)
- [An Evolutionary, Gradient-Free, Query-Efficient, Black-Box Algorithm for Generating Adversarial Instances in Deep Networks](https://mlsanity.com/paper/2208.08297/) (2022)
- [Enabling Uncertainty Estimation in Iterative Neural Networks](https://mlsanity.com/paper/2403.16732/) (2024)
- [Physics-Informed Neural Network Method for Solving One-Dimensional Advection Equation Using PyTorch](https://mlsanity.com/paper/2103.09662/) (2021)
- [SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruning](https://mlsanity.com/paper/2112.04002/) (2021)
- [Correlation-Embedded Transformer Tracking: A Single-Branch Framework](https://mlsanity.com/paper/2401.12743/) (2024)
- [Epistemic Uncertainty in Conformal Scores: A Unified Approach](https://mlsanity.com/paper/2502.06995/) (2025)
- [Progress Measures for Grokking on Real-world Tasks](https://mlsanity.com/paper/2405.12755/) (2024)
- [No More Adam: Learning Rate Scaling at Initialization is All You Need](https://mlsanity.com/paper/2412.11768/) (2024)
- [Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networks](https://mlsanity.com/paper/2103.17174/) (2021)
- [Kolmogorov-Arnold Recurrent Network for Short Term Load Forecasting Across Diverse Consumers](https://mlsanity.com/paper/2501.06965/) (2025)
- [Data-Driven Short-Term Daily Operational Sea Ice Regional Forecasting](https://mlsanity.com/paper/2210.08877/) (2022)
- [Leveraging Operator Learning to Accelerate Convergence of the Preconditioned Conjugate Gradient Method](https://mlsanity.com/paper/2508.00101/) (2025)
- [Transfer Learning Enhanced Full Waveform Inversion](https://mlsanity.com/paper/2302.11259/) (2023)
- [Structured state-space models are deep Wiener models](https://mlsanity.com/paper/2312.06211/) (2023)
- [Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?](https://mlsanity.com/paper/2505.12871/) (2025)
- [Model-Driven Deep Learning for Non-Coherent Massive Machine-Type Communications](https://mlsanity.com/paper/2301.00516/) (2023)
- [Data Driven Reward Initialization for Preference based Reinforcement Learning](https://mlsanity.com/paper/2302.08733/) (2023)

## Abstract (arXiv)

A proper initialization of the weights in a neural network is critical to its convergence. Current insights into weight initialization come primarily from linear activation functions. In this paper, I develop a theory for weight initializations with non-linear activations. First, I derive a general weight initialization strategy for any neural network using activation functions differentiable at 0. Next, I derive the weight initialization strategy for the Rectified Linear Unit (RELU), and provide theoretical insights into why the Xavier initialization is a poor choice with RELU activations. My analysis provides a clear demonstration of the role of non-linearities in determining the proper weight initializations.