Initializing ReLU networks in an expressive subspace of weights
Not on the map. Why?
Abstract (arXiv)
Using a mean-field theory of signal propagation, we analyze the evolution of correlations between two signals propagating forward through a deep ReLU network with correlated weights. Signals become highly correlated in deep ReLU networks with uncorrelated weights. We show that ReLU networks with anti-correlated weights can avoid this fate and have a chaotic phase where the signal correlations saturate below unity. Consistent with this analysis, we find that networks initialized with anti-correlated weights can train faster (in a teacher-student setting) by taking advantage of the increased expressivity in the chaotic phase. Combining this with a previously proposed strategy of using an asymmetric initialization to reduce dead node probability, we propose an initialization scheme that allows faster training and learning than the best-known initializations.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 54 references that are in the map. First posted on arXiv 23 Mar 2021.
Builds on
32 of 54 references.
References · 32
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2016Layer Normalizationcited by 5,579
- 2014Going Deeper with Convolutionscited by 4,231
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2018Group Normalizationcited by 1,514
- 2018On the Spectral Bias of Neural Networkscited by 1,321
- 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
- 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
20 more
- 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2017Self-Normalizing Neural Networkscited by 591
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
- 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
- 2016On the Expressive Power of Deep Neural Networkscited by 302
- 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
- 2016Deep Information Propagationcited by 194
- 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
- 2015Training Very Deep Networkscited by 172
- 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
- 2015All you need is a good initcited by 124
- 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
- 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
- 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
- 2018Gradient Descent Quantizes ReLU Network Featurescited by 57
- 2016Parametric Exponential Linear Unit for Deep Convolutional Neural Networkscited by 32
- 2018On the Selection of Initialization and Activation Function for Deep Neural Networkscited by 11
- 2017Exploring the Function Space of Deep-Learning Machinescited by 7
- 2019Large Deviation Analysis of Function Sensitivity in Random Deep Neural Networkscited by 2
Built on by
Citing papers per year: 2023: 1.
1 paper.