mlsanityData to 14 Oct 2026

Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks

arXiv 2309.15244 ↗First posted 26 Sep 2023cs.LG math.OCcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a novel training approach called the Homotopy Relaxation Training Algorithm (HRTA), aimed at accelerating the training process in contrast to traditional methods. Our algorithm incorporates two key mechanisms: one involves building a homotopy activation function that seamlessly connects the linear activation function with the ReLU activation function; the other technique entails relaxing the homotopy parameter to enhance the training refinement process. We have conducted an in-depth analysis of this novel method within the context of the neural tangent kernel (NTK), revealing significantly improved convergence rates. Our experimental results, especially when considering networks with larger widths, validate the theoretical conclusions. This proposed HRTA exhibits the potential for other activation functions and deep neural networks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 29 of its 57 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

29 of 57 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  7. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  8. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  9. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 583
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  11. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
  12. 2017Sobolev Training for Neural Networkscited by 143
17 more
  1. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 67
  2. 2014Learning Activation Functions to Improve Deep Neural Networkscited by 63
  3. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  4. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  5. 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
  6. 2021Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functionscited by 49
  7. 2018Scaling limit of the Stein variational gradient descent: the mean field regimecited by 35
  8. 2019A type of generalization error induced by initialization in deep neural networkscited by 33
  9. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  10. 2023Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivativescited by 30
  11. 2022On the Activation Function Dependence of the Spectral Bias of Neural Networkscited by 28
  12. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17
  13. 2020Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentumcited by 8
  14. 2019A homotopy training algorithm for fully connected neural networkscited by 7
  15. 2020ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performancecited by 7
  16. 2019Accelerated Optimization With Orthogonality Constraintscited by 6
  17. 2022Side Effects of Learning from Low-dimensional Data Embedded in a Euclidean Spacecited by 6

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equationscited by 9
  2. 2024Quantifying Training Difficulty and Accelerating Convergence in Neural Network-Based PDE Solverscited by 4
  3. 2025Learn Singularly Perturbed Solutions via Homotopy Dynamicscited by 2
  4. 2025Statistical Learning Guarantees for Group-Invariant Barron Functionscited by 1