# Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Framework

arXiv: https://arxiv.org/abs/2009.02868 · HTML: https://mlsanity.com/paper/2009.02868/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2020.

## Built on by (most cited first, 9 of 9)

- [Convergence Rates of Training Deep Neural Networks via Alternating Minimization Methods](https://mlsanity.com/paper/2208.14318/) (2022)
- [A Convergent ADMM Framework for Efficient Neural Network Training](https://mlsanity.com/paper/2112.11619/) (2021)
- [Learning with Local Gradients at the Edge](https://mlsanity.com/paper/2208.08503/) (2022)
- [A Triple-Inertial Accelerated Alternating Optimization Method for Deep Learning Training](https://mlsanity.com/paper/2503.08489/) (2025)
- [Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework](https://mlsanity.com/paper/2105.09837/) (2021)
- [Community-based Layerwise Distributed Training of Graph Convolutional Networks](https://mlsanity.com/paper/2112.09335/) (2021)
- [ADMM Algorithms for Residual Network Training: Convergence Analysis and Parallel Implementation](https://mlsanity.com/paper/2310.15334/) (2023)
- [ADMM-Based Training for Spiking Neural Networks](https://mlsanity.com/paper/2505.05527/) (2025)
- [Neural Network Training via Stochastic Alternating Minimization with Trainable Step Sizes](https://mlsanity.com/paper/2508.04193/) (2025)

## Abstract (arXiv)

Alternating Direction Method of Multipliers (ADMM) has recently been proposed as a potential alternative optimizer to the Stochastic Gradient Descent(SGD) for deep learning problems. This is because ADMM can solve gradient vanishing and poor conditioning problems. Moreover, it has shown good scalability in many large-scale deep learning applications. However, there still lacks a parallel ADMM computational framework for deep neural networks because of layer dependency among variables. In this paper, we propose a novel parallel deep learning ADMM framework (pdADMM) to achieve layer parallelism: parameters in each layer of neural networks can be updated independently in parallel. The convergence of the proposed pdADMM to a critical point is theoretically proven under mild conditions. The convergence rate of the pdADMM is proven to be o(1/k) where k is the number of iterations. Extensive experiments on six benchmark datasets demonstrated that our proposed pdADMM can lead to more than 10 times speedup for training large-scale deep neural networks, and outperformed most of the comparison methods. Our code is available at: https://github.com/xianggebenben/pdADMM.