mlsanityData to 14 Oct 2026

IDInit: A Universal and Stable Initialization Method for Neural Network Training

arXiv 2503.04626 ↗First posted 6 Mar 2025cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Deep neural networks have achieved remarkable accomplishments in practice. The success of these networks hinges on effective initialization methods, which are vital for ensuring stable and rapid convergence during training. Recently, initialization methods that maintain identity transition within layers have shown good efficiency in network training. These techniques (e.g., Fixup) set specific weights to zero to achieve identity control. However, settings of remaining weight (e.g., Fixup uses random values to initialize non-zero weights) will affect the inductive bias that is achieved only by a zero weight, which may be harmful to training. Addressing this concern, we introduce fully identical initialization (IDInit), a novel method that preserves identity in both the main and sub-stem layers of residual networks. IDInit employs a padded identity-like matrix to overcome rank constraints in non-square weight matrices. Furthermore, we show the convergence problem of an identity matrix can be solved by stochastic gradient descent. Additionally, we enhance the universality of IDInit by processing higher-order weights and addressing dead neuron problems. IDInit is a straightforward yet effective initialization method, with improved convergence, stability, and performance across various settings, including large-scale datasets and deep models.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

35 of 45 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  6. 2017Squeeze-and-Excitation Networkscited by 4,789
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
23 more
  1. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  2. 2019GhostNet: More Features from Cheap Operationscited by 410
  3. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  4. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  5. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  6. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  7. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  8. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  9. 2016Identity Matters in Deep Learningcited by 109
  10. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  11. 2017DiracNets: Training Very Deep Neural Networks Without Skip-Connectionscited by 47
  12. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  13. 2020Deep Isometric Learning for Visual Recognitioncited by 39
  14. 2019How to Initialize your Network? Robust Initialization for WeightNorm & ResNetscited by 36
  15. 2023Tensor Networks Meet Neural Networks: A Survey and Future Perspectivescited by 32
  16. 2020Heuristic Rank Selection with Progressively Searching Tensor Ring Networkcited by 28
  17. 2021ZerO Initialization: Initializing Neural Networks with only Zeros and Onescited by 28
  18. 2018Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Functioncited by 22
  19. 2019Dynamical Isometry and a Mean Field Theory of LSTMs and GRUscited by 17
  20. 2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
  21. 2020Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?cited by 12
  22. 2024Preparing Lessons for Progressive Training on Language Modelscited by 10
  23. 2022A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networkscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025A Good Start Matters: Enhancing Continual Learning with Data-Driven Weight Initializationcited by 1
  2. 2025It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMscited by 0
  3. 2025Weights initialization of neural networks for function approximationcited by 0
  4. 2026SGR-OCC: Evolving Monocular Priors for Embodied 3D Occupancy Prediction via Soft-Gating Lifting and Semantic-Adaptive Geometric Refinementcited by 0
  5. 2026Bit-Identical Medical Deep Learning via Structured Orthogonal Initializationcited by 0