# Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methods

arXiv: https://arxiv.org/abs/2312.13677 · HTML: https://mlsanity.com/paper/2312.13677/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Training Very Deep Networks](https://mlsanity.com/paper/1507.06228/) (2015)
- [Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis](https://mlsanity.com/paper/1802.09941/) (2018)
- [ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!](https://mlsanity.com/paper/2202.09357/) (2022)
- [Deep learning with Elastic Averaging SGD](https://mlsanity.com/paper/1412.6651/) (2014)
- [Layer-Parallel Training of Deep Residual Neural Networks](https://mlsanity.com/paper/1812.04352/) (2018)
- [Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learning](https://mlsanity.com/paper/2207.04338/) (2022)
- [A Multi-Batch L-BFGS Method for Machine Learning](https://mlsanity.com/paper/1605.06049/) (2016)
- [Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies](https://mlsanity.com/paper/2306.17648/) (2023)
- [Quasi-Newton Methods for Machine Learning: Forget the Past, Just Sample](https://mlsanity.com/paper/1901.09997/) (2019)
- [Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance](https://mlsanity.com/paper/2007.07213/) (2020)
- [Globally Convergent Multilevel Training of Deep Residual Networks](https://mlsanity.com/paper/2107.07572/) (2021)
- [A Stochastic Trust Region Algorithm Based on Careful Step Normalization](https://mlsanity.com/paper/1712.10277/) (2017)
- [A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks](https://mlsanity.com/paper/2111.04949/) (2021)
- [On Solving L-SR1 Trust-Region Subproblems](https://mlsanity.com/paper/1506.07222/) (2015)
- [Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descent](https://mlsanity.com/paper/1812.00542/) (2018)
- [Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training](https://mlsanity.com/paper/2302.07049/) (2023)
- [On the Globalization of ASPIN Employing Trust-Region Control Strategies -- Convergence Analysis and Numerical Examples](https://mlsanity.com/paper/2104.05672/) (2021)
- [A Momentum Accelerated Adaptive Cubic Regularization Method for Nonconvex Optimization](https://mlsanity.com/paper/2210.05987/) (2022)

## Built on by (most cited first, 2 of 2)

- [Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimization](https://mlsanity.com/paper/2507.11513/) (2025)
- [A Non-Monotone Preconditioned Trust-Region Method for Neural Network Training](https://mlsanity.com/paper/2605.14860/) (2026)

## Abstract (arXiv)

We propose to train neural networks (NNs) using a novel variant of the ``Additively Preconditioned Trust-region Strategy'' (APTS). The proposed method is based on a parallelizable additive domain decomposition approach applied to the neural network's parameters. Built upon the TR framework, the APTS method ensures global convergence towards a minimizer. Moreover, it eliminates the need for computationally expensive hyper-parameter tuning, as the TR algorithm automatically determines the step size in each iteration. We demonstrate the capabilities, strengths, and limitations of the proposed APTS training method by performing a series of numerical experiments. The presented numerical study includes a comparison with widely used training methods such as SGD, Adam, LBFGS, and the standard TR method.