mlsanityData to 14 Oct 2026

Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis

arXiv 2410.02242 ↗First posted 3 Oct 2024cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

As a neural network's depth increases, it can improve generalization performance. However, training deep networks is challenging due to gradient and signal propagation issues. To address these challenges, extensive theoretical research and various methods have been introduced. Despite these advances, effective weight initialization methods for tanh neural networks remain insufficiently investigated. This paper presents a novel weight initialization method for neural networks with tanh activation function. Based on an analysis of the fixed points of the function tanh(ax), the proposed method aims to determine values of a that mitigate activation saturation. A series of experiments on various classification datasets and physics-informed neural networks demonstrates that the proposed method outperforms Xavier initialization methods~(with or without normalization) in terms of robustness across different network sizes, data efficiency, and convergence speed. Code is available at https://github.com/1HyunwooLee/Tanh-Init

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 29 of its 62 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

29 of 62 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Layer Normalizationcited by 5,669
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  6. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  7. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  9. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 608
  10. 2016Error bounds for approximations with deep ReLU networkscited by 574
  11. 2020B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Datacited by 335
  12. 2016On the Expressive Power of Deep Neural Networkscited by 306
17 more
  1. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  2. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  3. 2021Gradient-enhanced physics-informed neural networks for forward and inverse PDE problemscited by 222
  4. 2015All you need is a good initcited by 124
  5. 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 113
  6. 2021Parallel Physics-Informed Neural Networks via Domain Decompositioncited by 112
  7. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 96
  8. 2020Minimum Width for Universal Approximationcited by 94
  9. 2022Physics-informed neural networks for inverse problems in supersonic flowscited by 76
  10. 2020Physics-informed neural network for ultrasound nondestructive quantification of surface breaking crackscited by 68
  11. 2021A novel meta-learning initialization method for physics-informed neural networkscited by 35
  12. 2022Enhanced Physics-Informed Neural Networks with Augmented Lagrangian Relaxation Method (AL-PINNs)cited by 35
  13. 2021ZerO Initialization: Initializing Neural Networks with only Zeros and Onescited by 28
  14. 2021Approximation capability of two hidden layer feedforward neural networks with fixed weightscited by 24
  15. 2017On the approximation by single hidden layer feedforward neural networks with fixed weightscited by 19
  16. 2023MultiAdam: Parameter-wise Scale-invariant Optimizer for Multiscale Training of Physics-informed Neural Networkscited by 17
  17. 2023Improved weight initialization for deep and narrow feedforward neural networkcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 36
  2. 2026Discovering Data Manifold Geometry through Geometric Propertiescited by 1
  3. 2025Mask-PINNs: Mitigating Internal Covariate Shift in Physics-Informed Neural Networkscited by 0
  4. 2025Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networkscited by 0
  5. 2025Beyond Gaussian Initializations: Signal Preserving Weight Initialization for Odd-Sigmoid Activationscited by 0