AutoInit: Automatic Initialization via Jacobian Tuning
Not on the map. Why?
Abstract (arXiv)
Good initialization is essential for training Deep Neural Networks (DNNs). Oftentimes such initialization is found through a trial and error approach, which has to be applied anew every time an architecture is substantially modified, or inherited from smaller size networks leading to sub-optimal initialization. In this work we introduce a new and cheap algorithm, that allows one to find a good initialization automatically, for general feed-forward DNNs. The algorithm utilizes the Jacobian between adjacent network blocks to tune the network hyperparameters to criticality. We solve the dynamics of the algorithm for fully connected networks with ReLU and derive conditions for its convergence. We then extend the discussion to more general architectures with BatchNorm and residual connections. Finally, we apply our method to ResMLP and VGG architectures, where the automatic one-shot initialization found by our method shows good performance on vision tasks.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 27 Jun 2022.
Builds on
30 of 37 references.
References · 30
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
- 2018Group Normalizationcited by 1,524
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
- 2021Going deeper with Image Transformerscited by 764
- 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
- 2017Deep Neural Networks as Gaussian Processescited by 622
- 2017Self-Normalizing Neural Networkscited by 598
18 more
- 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 339
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
- 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
- 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
- 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
- 2015Training Very Deep Networkscited by 173
- 2021The Principles of Deep Learning Theorycited by 173
- 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
- 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
- 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
- 2019Robust Learning with Jacobian Regularizationcited by 129
- 2015All you need is a good initcited by 124
- 2020Infinite attention: NNGP and NTK for deep attention networkscited by 121
- 2018The Emergence of Spectral Universality in Deep Networkscited by 74
- 2019A Mean Field Theory of Batch Normalizationcited by 58
- 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 52
- 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
- 2021Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applicationscited by 8
Built on by
Citing papers per year: 2023: 2; 2024: 1.
3 papers.