Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methods
Not on the map. Why?
Abstract (arXiv)
We propose to train neural networks (NNs) using a novel variant of the ``Additively Preconditioned Trust-region Strategy'' (APTS). The proposed method is based on a parallelizable additive domain decomposition approach applied to the neural network's parameters. Built upon the TR framework, the APTS method ensures global convergence towards a minimizer. Moreover, it eliminates the need for computationally expensive hyper-parameter tuning, as the TR algorithm automatically determines the step size in each iteration. We demonstrate the capabilities, strengths, and limitations of the proposed APTS training method by performing a series of numerical experiments. The presented numerical study includes a comparison with widely used training methods such as SGD, Adam, LBFGS, and the standard TR method.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 47 references that are in the map. First posted on arXiv 21 Dec 2023.
Builds on
23 of 47 references.
References · 23
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
- 2016An overview of gradient descent optimization algorithmscited by 1,021
- 2019On the Convergence of Adam and Beyondcited by 834
- 2015Training Very Deep Networkscited by 172
- 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 154
- 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
- 2014Deep learning with Elastic Averaging SGDcited by 90
- 2018Layer-Parallel Training of Deep Residual Neural Networkscited by 35
- 2022Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learningcited by 33
- 2016A Multi-Batch L-BFGS Method for Machine Learningcited by 30
11 more
- 2023Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategiescited by 20
- 2019Quasi-Newton Methods for Machine Learning: Forget the Past, Just Samplecited by 12
- 2020Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidancecited by 11
- 2021Globally Convergent Multilevel Training of Deep Residual Networkscited by 11
- 2017A Stochastic Trust Region Algorithm Based on Careful Step Normalizationcited by 10
- 2021A Survey and Empirical Evaluation of Parallel Deep Learning Frameworkscited by 7
- 2015On Solving L-SR1 Trust-Region Subproblemscited by 6
- 2018Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descentcited by 6
- 2023Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Trainingcited by 6
- 2021On the Globalization of ASPIN Employing Trust-Region Control Strategies -- Convergence Analysis and Numerical Examplescited by 3
- 2022A Momentum Accelerated Adaptive Cubic Regularization Method for Nonconvex Optimizationcited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.