mlsanityData to 7 Oct 2026

On the Impact of the Activation Function on Deep Neural Networks Training

arXiv 1902.06853 ↗First posted 19 Feb 2019stat.ML cs.AI cs.LGcited by 101

Not on the map. Why?

Abstract (arXiv)

The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propagation. Understanding the theoretical properties of untrained random networks is key to identifying which deep networks may be trained successfully as recently demonstrated by Samuel et al (2017) who showed that for deep feedforward neural networks only a specific choice of hyperparameters known as the `Edge of Chaos' can lead to good performance. While the work by Samuel et al (2017) discuss trainability issues, we focus here on training acceleration and overall performance. We give a comprehensive theoretical analysis of the Edge of Chaos and show that we can indeed tune the initialization parameters and the activation function in order to accelerate the training and improve the performance.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 15; 2023: 18; 2024: 13; 2025: 18; 2026: 14.

101 papers.

Sort:

By area:

  1. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  2. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  3. 2021Gradients are Not All You Needcited by 82
  4. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  5. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 65
  6. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  7. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 51
  8. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  9. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  10. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  11. 2023Simplifying Transformer Blockscited by 37
  12. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
38 more
  1. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  2. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  3. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  4. 2023Leave-one-out Distinguishability in Machine Learningcited by 20
  5. 2022Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learningcited by 19
  6. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  7. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  8. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  9. 2021Truly Sparse Neural Networks at Scalecited by 15
  10. 2021Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspectivecited by 14
  11. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  12. 2021Activation function design for deep networks: linearity and effective initialisationcited by 13
  13. 2025PLoP: Precise LoRA Placement for Efficient Finetuning of Large Modelscited by 13
  14. 2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 13
  15. 2021Finite volume method network for acceleration of unsteady computational fluid dynamics: non-reacting and reacting flowscited by 12
  16. 2022Wide and Deep Neural Networks Achieve Optimality for Classificationcited by 12
  17. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  18. 2021Wide stochastic networks: Gaussian limit and PAC-Bayesian trainingcited by 10
  19. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10
  20. 2021Automatic Fault Detection for Deep Learning Programs Using Graph Transformationscited by 9
  21. 2021Neuron Campaign for Initialization Guided by Information Bottleneck Theorycited by 9
  22. 2022Gaussian Pre-Activations in Neural Networks: Myth or Reality?cited by 9
  23. 2024Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimizationcited by 9
  24. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertisecited by 9
  25. 2021Deep Stable neural networks: large-width asymptotics and convergence ratescited by 8
  26. 2025Physics of Skill Learningcited by 8
  27. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  28. 2026Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuningcited by 8
  29. 2021Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networkscited by 7
  30. 2021Regularization in ResNet with Stochastic Depthcited by 7
  31. 2021Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metricscited by 7
  32. 2022Analysis of Convolutions, Non-linearity and Depth in Graph Neural Networks using Neural Tangent Kernelcited by 7
  33. 2021Bayesian neural network unit priors and generalized Weibull-tail propertycited by 5
  34. 2022Self-scalable Tanh (Stan): Faster Convergence and Better Generalization in Physics-informed Neural Networkscited by 5
  35. 2023On the Initialisation of Wide Low-Rank Feedforward Neural Networkscited by 5
  36. 2024Task structure and nonlinearity jointly determine learned representational geometrycited by 5
  37. 2024Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptronscited by 5
  38. 2021Universal scaling laws in the gradient descent training of neural networkscited by 4

50 most cited of 101 All 101 (JSON)