mlsanityData to 14 Oct 2026

ZerO Initialization: Initializing Neural Networks with only Zeros and Ones

arXiv 2110.12661 ↗First posted 25 Oct 2021cs.LG cs.CVcited by 28

Not on the map. Why?

Abstract (arXiv)

Deep neural networks are usually initialized with random weights, with adequately selected initial variance to ensure stable signal propagation during training. However, selecting the appropriate variance becomes challenging especially as the number of layers grows. In this work, we replace random weight initialization with a fully deterministic initialization scheme, viz., ZerO, which initializes the weights of networks with only zeros and ones (up to a normalization factor), based on identity and Hadamard transforms. Through both theoretical and empirical studies, we demonstrate that ZerO is able to train networks without damaging their expressivity. Applying ZerO on ResNet achieves state-of-the-art performance on various datasets, including ImageNet, which suggests random weights may be unnecessary for network initialization. In addition, ZerO has many benefits, such as training ultra deep networks (without batch-normalization), exhibiting low-rank learning trajectories that result in low-rank and sparse solutions, and improving training reproducibility.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 25 Oct 2021.

Builds on

22 of 30 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  7. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  8. 2017Convolutional Sequence to Sequence Learningcited by 552
  9. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  10. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  11. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  12. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
10 more
  1. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 162
  2. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 157
  3. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  4. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  5. 2016Identity Matters in Deep Learningcited by 109
  6. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  7. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  8. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  9. 2021Implicit Regularization in Tensor Factorizationcited by 44
  10. 2020Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?cited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 6; 2025: 13; 2026: 4.

28 papers.

Sort:

By area:

  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  2. 2023Efficient Large Language Models: A Surveycited by 186
  3. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  4. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  5. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 19
  6. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 17
  7. 2025Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelismcited by 12
  8. 2025Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learningcited by 12
  9. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
  10. 2023Improved weight initialization for deep and narrow feedforward neural networkcited by 7
  11. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  12. 2024Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysiscited by 5
16 more
  1. 2025Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Modelscited by 5
  2. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 5
  3. 2025IDInit: A Universal and Stable Initialization Method for Neural Network Trainingcited by 5
  4. 2025Sinusoidal Initialization, Time for a New Startcited by 5
  5. 2025Conda: Column-Normalized Adam for Training Large Language Models Fastercited by 5
  6. 2022Hypercomplex Image-to-Image Translationcited by 4
  7. 2023QuickSRNet: Plain Single-Image Super-Resolution Architecture for Faster Inference on Mobile Platformscited by 4
  8. 2024Discrete Cosine Transform Based Decorrelated Attention for Vision Transformerscited by 2
  9. 2024An Effective Weight Initialization Method for Deep Learning: Application to Satellite Image Classificationcited by 1
  10. 2025A Good Start Matters: Enhancing Continual Learning with Data-Driven Weight Initializationcited by 1
  11. 2025MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptationcited by 1
  12. 2026Optimal Projection-Free Adaptive SGD for Matrix Optimizationcited by 1
  13. 2025SWiFT: Soft-Mask Weight Fine-tuning for Bias Mitigationcited by 0
  14. 2026LION-DG: Layer-Informed Initialization with Deep Gradient Protocols for Accelerated Neural Network Trainingcited by 0
  15. 2026Bit-Identical Medical Deep Learning via Structured Orthogonal Initializationcited by 0
  16. 2026Procedural Core: A Compact Recurrent Initialization for Vision Transformerscited by 0