mlsanityData to 14 Oct 2026

Convergence Rates of Training Deep Neural Networks via Alternating Minimization Methods

arXiv 2208.14318 ↗First posted 30 Aug 2022cs.LG math.OCcited by 4

Not on the map. Why?

Abstract (arXiv)

Training deep neural networks (DNNs) is an important and challenging optimization problem in machine learning due to its non-convexity and non-separable structure. The alternating minimization (AM) approaches split the composition structure of DNNs and have drawn great interest in the deep learning and optimization communities. In this paper, we propose a unified framework for analyzing the convergence rate of AM-type network training methods. Our analysis is based on the non-monotone j-step sufficient decrease conditions and the Kurdyka-Lojasiewicz (KL) property, which relaxes the requirement of designing descent algorithms. We show the detailed local convergence rate if the KL exponent θ varies in [0,1). Moreover, the local R-linear convergence is discussed under a stronger j-step sufficient decrease condition.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 22 of its 49 references that are in the map. First posted on arXiv 30 Aug 2022.

Builds on

22 of 49 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  5. 2016Wide & Deep Learning for Recommender Systemscited by 747
  6. 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
  7. 2018Global Convergence of Block Coordinate Descent in Deep Learningcited by 38
  8. 2016Calculus of the exponent of Kurdyka-Lojasiewicz inequality and its applications to linear convergence of first-order methodscited by 35
  9. 2014Splitting methods with variable metric for KL functionscited by 26
  10. 2017Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networkscited by 26
  11. 2018Beyond Backprop: Online Alternating Minimization with Auxiliary Variablescited by 23
  12. 2019ADMM for Efficient Deep Learning with Global Convergencecited by 20
10 more
  1. 2018Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Trainingcited by 19
  2. 2016A Primer on Coordinate Descent Algorithmscited by 17
  3. 2021Convergence of Random Reshuffling Under The Kurdyka-Lojasiewicz Inequalitycited by 16
  4. 2014Douglas-Rachford splitting for nonconvex optimization with application to nonconvex feasibility problemscited by 15
  5. 2018A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Trainingcited by 15
  6. 2019On ADMM in Deep Learning: Convergence and Saturation-Avoidancecited by 15
  7. 2020Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Frameworkcited by 9
  8. 2018Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimizationcited by 5
  9. 2016Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networkscited by 4
  10. 2018Learning ReLU Networks via Alternating Minimizationcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 0; 2025: 2.

4 papers.

  1. 2023Visual Attention Prompted Prediction and Learningcited by 2
  2. 2023ADMM Algorithms for Residual Network Training: Convergence Analysis and Parallel Implementationcited by 0
  3. 2025Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Conditioncited by 0
  4. 2025Block Coordinate Descent for Neural Networks Provably Finds Global Minimacited by 0