# ADMM for Efficient Deep Learning with Global Convergence

arXiv: https://arxiv.org/abs/1905.13611 · HTML: https://mlsanity.com/paper/1905.13611/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2019.

## Built on by (most cited first, 20 of 20)

- [Efficient Global Optimization of Two-Layer ReLU Networks: Quadratic-Time Algorithms and Adversarial Training](https://mlsanity.com/paper/2201.01965/) (2022)
- [Physics-Informed Deep Learning For Traffic State Estimation: A Survey and the Outlook](https://mlsanity.com/paper/2303.02063/) (2023)
- [Curriculum Learning for Graph Neural Networks: Which Edges Should We Learn First](https://mlsanity.com/paper/2310.18735/) (2023)
- [Convergence Rates of Training Deep Neural Networks via Alternating Minimization Methods](https://mlsanity.com/paper/2208.14318/) (2022)
- [Inertial Proximal Deep Learning Alternating Minimization for Efficient Neutral Network Training](https://mlsanity.com/paper/2102.00267/) (2021)
- [Stochastic Block-ADMM for Training Deep Networks](https://mlsanity.com/paper/2105.00339/) (2021)
- [Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation](https://mlsanity.com/paper/2308.12053/) (2023)
- [BADM: Batch ADMM for Deep Learning](https://mlsanity.com/paper/2407.01640/) (2024)
- [Online Parallel Multi-Task Relationship Learning via Alternating Direction Method of Multipliers](https://mlsanity.com/paper/2411.06135/) (2024)
- [Preconditioned Inexact Stochastic ADMM for Deep Model](https://mlsanity.com/paper/2502.10784/) (2025)
- [Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variables](https://mlsanity.com/paper/2504.21501/) (2025)
- [Learning with Local Gradients at the Edge](https://mlsanity.com/paper/2208.08503/) (2022)
- [A Triple-Inertial Accelerated Alternating Optimization Method for Deep Learning Training](https://mlsanity.com/paper/2503.08489/) (2025)
- [Augmented Lagrangian Predictive Coding](https://mlsanity.com/paper/2605.31022/) (2026)
- [Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework](https://mlsanity.com/paper/2105.09837/) (2021)
- [Community-based Layerwise Distributed Training of Graph Convolutional Networks](https://mlsanity.com/paper/2112.09335/) (2021)
- [ADMM Algorithms for Residual Network Training: Convergence Analysis and Parallel Implementation](https://mlsanity.com/paper/2310.15334/) (2023)
- [Optimizing ADMM and Over-Relaxed ADMM Parameters for Linear Quadratic Problems](https://mlsanity.com/paper/2401.00657/) (2024)
- [ADMM-Based Training for Spiking Neural Networks](https://mlsanity.com/paper/2505.05527/) (2025)
- [Neural Network Training via Stochastic Alternating Minimization with Trainable Step Sizes](https://mlsanity.com/paper/2508.04193/) (2025)

## Abstract (arXiv)

Alternating Direction Method of Multipliers (ADMM) has been used successfully in many conventional machine learning applications and is considered to be a useful alternative to Stochastic Gradient Descent (SGD) as a deep learning optimizer. However, as an emerging domain, several challenges remain, including 1) The lack of global convergence guarantees, 2) Slow convergence towards solutions, and 3) Cubic time complexity with regard to feature dimensions. In this paper, we propose a novel optimization framework for deep learning via ADMM (dlADMM) to address these challenges simultaneously. The parameters in each layer are updated backward and then forward so that the parameter information in each layer is exchanged efficiently. The time complexity is reduced from cubic to quadratic in (latent) feature dimensions via a dedicated algorithm design for subproblems that enhances them utilizing iterative quadratic approximations and backtracking. Finally, we provide the first proof of global convergence for an ADMM-based method (dlADMM) in a deep neural network problem under mild conditions. Experiments on benchmark datasets demonstrated that our proposed dlADMM algorithm outperforms most of the comparison methods.