mlsanityData to 13 Oct 2026

Gaussian Pre-Activations in Neural Networks: Myth or Reality?

arXiv 2205.12379 ↗First posted 24 May 2022cs.LG stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

The study of feature propagation at initialization in neural networks lies at the root of numerous initialization designs. An assumption very commonly made in the field states that the pre-activations are Gaussian. Although this convenient Gaussian hypothesis can be justified when the number of neurons per layer tends to infinity, it is challenged by both theoretical and experimental works for finite-width neural networks. Our major contribution is to construct a family of pairs of activation functions and initialization distributions that ensure that the pre-activations remain Gaussian throughout the network's depth, even in narrow neural networks. In the process, we discover a set of constraints that a neural network should fulfill to ensure Gaussian pre-activations. Additionally, we provide a critical review of the claims of the Edge of Chaos line of works and build an exact Edge of Chaos analysis. We also propose a unified view on pre-activations propagation, encompassing the framework of several well-known initialization procedures. Finally, our work provides a principled framework for answering the much-debated question: is it desirable to initialize the training of a neural network whose pre-activations are ensured to be Gaussian? Our code is available on GitHub: https://github.com/p-wol/gaussian-preact/ .

Key facts. Cited by 9 papers in the map (data through 13 Oct 2026). Builds on 31 of its 57 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

31 of 57 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,601
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,610
  4. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,225
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 786
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 737
  7. 2017Deep Neural Networks as Gaussian Processescited by 618
  8. 2017Self-Normalizing Neural Networkscited by 594
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 573
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 565
  12. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 497
19 more
  1. 2019Implicit Regularization in Deep Matrix Factorizationcited by 404
  2. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 316
  3. 2016Exponential expressivity in deep neural networks through transient chaoscited by 259
  4. 2016Deep Information Propagationcited by 194
  5. 2021Bayesian Neural Network Priors Revisitedcited by 121
  6. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  7. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  8. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  9. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  10. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 52
  11. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  12. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  13. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  14. 2017Online Natural Gradient as a Kalman Filtercited by 25
  15. 2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 24
  16. 2020Stable behaviour of infinitely wide deep neural networkscited by 23
  17. 2020Least squares binary quantization of neural networkscited by 21
  18. 2021Neuron Campaign for Initialization Guided by Information Bottleneck Theorycited by 9
  19. 2021Bayesian neural network unit priors and generalized Weibull-tail propertycited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 3; 2024: 1; 2025: 2; 2026: 3.

9 papers.

  1. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  2. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  3. 2024Streamlining Prediction in Bayesian Deep Learningcited by 5
  4. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  5. 2025VeLU: Variance-enhanced Learning Unit for Deep Neural Networkscited by 1
  6. 2025It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMscited by 0
  7. 2026Adaptive Label Error Detection: A Bayesian Approach to Mislabeled Data Detectioncited by 0
  8. 2026Batch Size or Negatives? A Selection Rule for Memory-Constrained Recommender Trainingcited by 0
  9. 2026Stable initialization without the CLTcited by 0