# Convergence Rates of Training Deep Neural Networks via Alternating Minimization Methods

arXiv: https://arxiv.org/abs/2208.14318 · HTML: https://mlsanity.com/paper/2208.14318/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 30 Aug 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Training Neural Networks Without Gradients: A Scalable ADMM Approach](https://mlsanity.com/paper/1605.02026/) (2016)
- [Global Convergence of Block Coordinate Descent in Deep Learning](https://mlsanity.com/paper/1803.00225/) (2018)
- [Calculus of the exponent of Kurdyka-Lojasiewicz inequality and its applications to linear convergence of first-order methods](https://mlsanity.com/paper/1602.02915/) (2016)
- [Splitting methods with variable metric for KL functions](https://mlsanity.com/paper/1405.1357/) (2014)
- [Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks](https://mlsanity.com/paper/1711.07354/) (2017)
- [Beyond Backprop: Online Alternating Minimization with Auxiliary Variables](https://mlsanity.com/paper/1806.09077/) (2018)
- [ADMM for Efficient Deep Learning with Global Convergence](https://mlsanity.com/paper/1905.13611/) (2019)
- [Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training](https://mlsanity.com/paper/1811.08039/) (2018)
- [A Primer on Coordinate Descent Algorithms](https://mlsanity.com/paper/1610.00040/) (2016)
- [Convergence of Random Reshuffling Under The Kurdyka-Lojasiewicz Inequality](https://mlsanity.com/paper/2110.04926/) (2021)
- [Douglas-Rachford splitting for nonconvex optimization with application to nonconvex feasibility problems](https://mlsanity.com/paper/1409.8444/) (2014)
- [A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Training](https://mlsanity.com/paper/1803.09082/) (2018)
- [On ADMM in Deep Learning: Convergence and Saturation-Avoidance](https://mlsanity.com/paper/1902.02060/) (2019)
- [Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Framework](https://mlsanity.com/paper/2009.02868/) (2020)
- [Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization](https://mlsanity.com/paper/1811.04187/) (2018)
- [Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks](https://mlsanity.com/paper/1611.01590/) (2016)
- [Learning ReLU Networks via Alternating Minimization](https://mlsanity.com/paper/1806.07863/) (2018)

## Built on by (most cited first, 4 of 4)

- [Visual Attention Prompted Prediction and Learning](https://mlsanity.com/paper/2310.08420/) (2023)
- [ADMM Algorithms for Residual Network Training: Convergence Analysis and Parallel Implementation](https://mlsanity.com/paper/2310.15334/) (2023)
- [Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Condition](https://mlsanity.com/paper/2505.08262/) (2025)
- [Block Coordinate Descent for Neural Networks Provably Finds Global Minima](https://mlsanity.com/paper/2510.22667/) (2025)

## Abstract (arXiv)

Training deep neural networks (DNNs) is an important and challenging optimization problem in machine learning due to its non-convexity and non-separable structure. The alternating minimization (AM) approaches split the composition structure of DNNs and have drawn great interest in the deep learning and optimization communities. In this paper, we propose a unified framework for analyzing the convergence rate of AM-type network training methods. Our analysis is based on the non-monotone j-step sufficient decrease conditions and the Kurdyka-Lojasiewicz (KL) property, which relaxes the requirement of designing descent algorithms. We show the detailed local convergence rate if the KL exponent θ varies in [0,1). Moreover, the local R-linear convergence is discussed under a stronger j-step sufficient decrease condition.