mlsanityData to 7 Oct 2026

How to Start Training: The Effect of Initialization and Architecture

arXiv 1803.01719 ↗First posted 5 Mar 2018stat.ML cs.LGcited by 109

Not on the map. Why?

Abstract (arXiv)

We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a symmetric distribution with variance 2/fan-in and, for ResNets, by correctly weighting the residual modules. We prove that the second failure mode, exponentially large variance of activation length, never occurs in residual nets once the first failure mode is avoided. In contrast, for fully connected nets, we prove that this failure mode can happen and is avoided by keeping constant the sum of the reciprocals of layer widths. We demonstrate empirically the effectiveness of our theoretical results in predicting when networks are able to start training. In particular, we note that many popular initializations fail our criteria, whereas correct initialization and architecture allows much deeper networks to be trained.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 37; 2022: 23; 2023: 23; 2024: 10; 2025: 10; 2026: 6.

109 papers.

Sort:

By area:

  1. 2021Multiscale Vision Transformerscited by 896
  2. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  3. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  4. 2021TransReID: Transformer-based Object Re-Identificationcited by 432
  5. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  6. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  7. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  8. 2021TokenPose: Learning Keypoint Tokens for Human Pose Estimationcited by 106
  9. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  10. 2021Proof-of-Learning: Definitions and Practicecited by 81
  11. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  12. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
38 more
  1. 2021SeMask: Semantically Masked Transformers for Semantic Segmentationcited by 55
  2. 2021A Survey on Green Deep Learningcited by 52
  3. 2022Rank Diminishing in Deep Neural Networkscited by 47
  4. 2021Fast Certified Robust Training with Short Warmupcited by 45
  5. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  6. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  7. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  8. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  9. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  10. 2023Hidden symmetries of ReLU networkscited by 38
  11. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  12. 2023Simplifying Transformer Blockscited by 37
  13. 2022How to Train Vision Transformer on Small-scale Datasets?cited by 34
  14. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  15. 2021KNAS: Green Neural Architecture Searchcited by 33
  16. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  17. 2023ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connectioncited by 25
  18. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 24
  19. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  20. 2024Parseval Regularization for Continual Reinforcement Learningcited by 21
  21. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  22. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  23. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  24. 2022Scaling ResNets in the Large-depth Regimecited by 18
  25. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  26. 2021"BNN - BN = ?": Training Binary Neural Networks without Batch Normalizationcited by 17
  27. 2023Rethinking Learning Rate Tuning in the Era of Large Language Modelscited by 16
  28. 2021Convolutional Hypercomplex Embeddings for Link Predictioncited by 15
  29. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 15
  30. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
  31. 2021Activation function design for deep networks: linearity and effective initialisationcited by 13
  32. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  33. 2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12
  34. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  35. 2023CS4ML: A general framework for active learning with arbitrary data based on Christoffel functionscited by 11
  36. 2023FedWon: Triumphing Multi-domain Federated Learning Without Normalizationcited by 11
  37. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  38. 2022Improving Fine-tuning of Self-supervised Models with Contrastive Initializationcited by 10

50 most cited of 109 All 109 (JSON)