mlsanityData to 14 Oct 2026

Expected Gradients of Maxout Networks and Consequences to Parameter Initialization

arXiv 2301.06956 ↗First posted 17 Jan 2023stat.ML cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We study the gradients of a maxout network with respect to inputs and parameters and obtain bounds for the moments depending on the architecture and the parameter distribution. We observe that the distribution of the input-output Jacobian depends on the input, which complicates a stable parameter initialization. Based on the moments of the gradients, we formulate parameter initialization strategies that avoid vanishing and exploding gradients in wide networks. Experiments with deep fully-connected and convolutional networks show that this strategy improves SGD and Adam training of deep maxout networks. In addition, we obtain refined bounds on the expected number of linear regions, results on the expected curve length distortion, and results on the NTK.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 58 references that are in the map. First posted on arXiv 17 Jan 2023.

Builds on

30 of 58 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  9. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  10. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  11. 2016On the Expressive Power of Deep Neural Networkscited by 306
  12. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
18 more
  1. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  2. 2016Benefits of depth in neural networkscited by 250
  3. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  4. 2013Maxout Networkscited by 187
  5. 2019Complexity of Linear Regions in Deep Networkscited by 167
  6. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
  7. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  8. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  9. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  10. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  11. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  12. 2015Representation Benefits of Deep Feedforward Networkscited by 86
  13. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  14. 2019Optimization for deep learning: theory and algorithmscited by 70
  15. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  16. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  17. 2021On the Expected Complexity of Maxout Networkscited by 15
  18. 2021Activation function design for deep networks: linearity and effective initialisationcited by 13

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9