mlsanityData to 7 Oct 2026

On weight initialization in deep neural networks

arXiv 1704.08863 ↗First posted 28 Apr 2017cs.LGcited by 67

Not on the map. Why?

Abstract (arXiv)

A proper initialization of the weights in a neural network is critical to its convergence. Current insights into weight initialization come primarily from linear activation functions. In this paper, I develop a theory for weight initializations with non-linear activations. First, I derive a general weight initialization strategy for any neural network using activation functions differentiable at 0. Next, I derive the weight initialization strategy for the Rectified Linear Unit (RELU), and provide theoretical insights into why the Xavier initialization is a poor choice with RELU activations. My analysis provides a clear demonstration of the role of non-linearities in determining the proper weight initializations.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 8; 2023: 15; 2024: 11; 2025: 14; 2026: 7.

67 papers.

Sort:

By area:

  1. 2023Efficient Large Language Models: A Surveycited by 183
  2. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 158
  3. 2021A Survey on Green Deep Learningcited by 52
  4. 2023Brain-inspired Computational Intelligence via Predictive Codingcited by 30
  5. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  6. 2021On Expressivity and Trainability of Quadratic Networkscited by 19
  7. 2025How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?cited by 19
  8. 2022Learning Invariant Weights in Neural Networkscited by 16
  9. 2023Disruptive Autoencoders: Leveraging Low-level features for 3D Medical Image Pre-trainingcited by 15
  10. 2021DEBOSH: Deep Bayesian Shape Optimizationcited by 14
  11. 2023Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignmentcited by 13
  12. 2024Automatically Learning Hybrid Digital Twins of Dynamical Systemscited by 13
38 more
  1. 2021MoRe-Fi: Motion-robust and Fine-grained Respiration Monitoring via Deep-Learning UWB Radarcited by 11
  2. 2022An Evolutionary, Gradient-Free, Query-Efficient, Black-Box Algorithm for Generating Adversarial Instances in Deep Networkscited by 11
  3. 2024Enabling Uncertainty Estimation in Iterative Neural Networkscited by 11
  4. 2021Physics-Informed Neural Network Method for Solving One-Dimensional Advection Equation Using PyTorchcited by 10
  5. 2021SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruningcited by 9
  6. 2024Correlation-Embedded Transformer Tracking: A Single-Branch Frameworkcited by 9
  7. 2025Epistemic Uncertainty in Conformal Scores: A Unified Approachcited by 9
  8. 2021Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networkscited by 7
  9. 2024Progress Measures for Grokking on Real-world Taskscited by 7
  10. 2025Kolmogorov-Arnold Recurrent Network for Short Term Load Forecasting Across Diverse Consumerscited by 7
  11. 2022Data-Driven Short-Term Daily Operational Sea Ice Regional Forecastingcited by 6
  12. 2024No More Adam: Learning Rate Scaling at Initialization is All You Needcited by 6
  13. 2025Leveraging Operator Learning to Accelerate Convergence of the Preconditioned Conjugate Gradient Methodcited by 6
  14. 2023Transfer Learning Enhanced Full Waveform Inversioncited by 4
  15. 2023Structured state-space models are deep Wiener modelscited by 4
  16. 2025Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?cited by 4
  17. 2023Model-Driven Deep Learning for Non-Coherent Massive Machine-Type Communicationscited by 3
  18. 2023Data Driven Reward Initialization for Preference based Reinforcement Learningcited by 3
  19. 2021A Weight Initialization Based on the Linear Product Structure for Neural Networkscited by 2
  20. 2021AdjointNet: Constraining machine learning models with physics-based codescited by 2
  21. 2025Deep Neural Networks Inspired by Differential Equationscited by 2
  22. 2022One-Bit Compressive Sensing: Can We Go Deep and Blind?cited by 1
  23. 2022Empirical Study of Overfitting in Deep FNN Prediction Models for Breast Cancer Metastasiscited by 1
  24. 2023The Broad Impact of Feature Imitation: Neural Enhancements Across Financial, Speech, and Physiological Domainscited by 1
  25. 2023SHOT: Suppressing the Hessian along the Optimization Trajectory for Gradient-Based Meta-Learningcited by 1
  26. 2023Energy-Efficient Power Control for Multiple-Task Split Inference in UAVs: A Tiny Learning-Based Approachcited by 1
  27. 2024Neural Conjugate Flows: Physics-informed architectures with flow structurecited by 1
  28. 2025Leveraging Hypernetworks and Learnable Kernels for Consumer Energy Forecasting Across Diverse Consumer Typescited by 1
  29. 2025VI3NR: Variance Informed Initialization for Implicit Neural Representationscited by 1
  30. 2026An Extensible and Lightweight Unified Architecture for Demosaicing Pixel-bin Image Sensorscited by 1
  31. 2026Mask-Based Priors Are More Persistent than Query-Key Initializationscited by 1
  32. 2021APS: A Large-Scale Multi-Modal Indoor Camera Positioning Systemcited by 0
  33. 2021Online-Learning Deep Neuro-Adaptive Dynamic Inversion Controller for Model Free Controlcited by 0
  34. 2021NFT-K: Non-Fungible Tangent Kernelscited by 0
  35. 2022Geometric Regularization from Overparameterizationcited by 0
  36. 2022Multilayer Perceptron Network Discriminates Larval Zebrafish Genotype using Behaviourcited by 0
  37. 2022Smooth Mathematical Function from Compact Neural Networkscited by 0
  38. 2023Detection and classification of vocal productions in large scale audio recordingscited by 0

50 most cited of 67 All 67 (JSON)