ADMM Algorithms for Residual Network Training: Convergence Analysis and Parallel Implementation
Not on the map. Why?
Abstract (arXiv)
We propose both serial and parallel proximal (linearized) alternating direction method of multipliers (ADMM) algorithms for training residual neural networks. In contrast to backpropagation-based approaches, our methods inherently mitigate the exploding gradient issue and are well-suited for parallel and distributed training through regional updates. Theoretically, we prove that the proposed algorithms converge at an R-linear (sublinear) rate for both the iteration points and the objective function values. These results hold without imposing stringent constraints on network width, depth, or training data size. Furthermore, we theoretically analyze our parallel/distributed ADMM algorithms, highlighting their reduced time complexity and lower per-node memory consumption. To facilitate practical deployment, we develop a control protocol for parallel ADMM implementation using Python's multiprocessing and interprocess communication. Experimental results validate the proposed ADMM algorithms, demonstrating rapid and stable convergence, improved performance, and high computational efficiency. Finally, we highlight the improved scalability and efficiency achieved by our parallel ADMM training strategy.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 23 Oct 2023.
Builds on
27 of 47 references.
References · 27
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
- 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
- 2015Global Convergence of ADMM in Nonconvex Nonsmooth Optimizationcited by 98
- 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
- 2019Comprehensive SNN Compression Using ADMM Optimization and Activity Regularizationcited by 45
15 more
- 2018Global Convergence of Block Coordinate Descent in Deep Learningcited by 38
- 2016Calculus of the exponent of Kurdyka-Lojasiewicz inequality and its applications to linear convergence of first-order methodscited by 35
- 2023New Bounds for Matrix Multiplication: from Alpha to Omegacited by 34
- 2018Beyond Backprop: Online Alternating Minimization with Auxiliary Variablescited by 23
- 2019ADMM for Efficient Deep Learning with Global Convergencecited by 20
- 2018Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Trainingcited by 19
- 2019On ADMM in Deep Learning: Convergence and Saturation-Avoidancecited by 15
- 2020Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Frameworkcited by 9
- 2021Local Critic Training for Model-Parallel Learning of Deep Neural Networkscited by 5
- 2021Linearly-constrained nonsmooth optimization for training autoencoderscited by 5
- 2022An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsitycited by 5
- 2015Alternating Direction Method of Multipliers for A Class of Nonconvex and Nonsmooth Problems with Applications to Background/Foreground Extractioncited by 4
- 2022Convergence Rates of Training Deep Neural Networks via Alternating Minimization Methodscited by 4
- 2014A Schur Complement Based Semi-Proximal ADMM for Convex Quadratic Conic Programming and Extensionscited by 3
- 2015A provably convergent alternating minimization method for mean field inferencecited by 1
Built on by
Not cited yet.