# Global Convergence of Block Coordinate Descent in Deep Learning

arXiv: https://arxiv.org/abs/1803.00225 · HTML: https://mlsanity.com/paper/1803.00225/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2018.

## Built on by (most cited first, 30 of 38)

- [On the Convergence of Muon and Beyond](https://mlsanity.com/paper/2509.15816/) (2025)
- [LocoProp: Enhancing BackProp via Local Loss Optimization](https://mlsanity.com/paper/2106.06199/) (2021)
- [Nonlinear Transform Induced Tensor Nuclear Norm for Tensor Completion](https://mlsanity.com/paper/2110.08774/) (2021)
- [0/1 Deep Neural Networks via Block Coordinate Descent](https://mlsanity.com/paper/2206.09379/) (2022)
- [BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blocks](https://mlsanity.com/paper/2406.17296/) (2024)
- [FedBCD:Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning](https://mlsanity.com/paper/2603.05116/) (2026)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Coordinate descent on the orthogonal group for recurrent neural network training](https://mlsanity.com/paper/2108.00051/) (2021)
- [An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity](https://mlsanity.com/paper/2205.05428/) (2022)
- [Convergent plug-and-play with proximal denoiser and unconstrained regularization parameter](https://mlsanity.com/paper/2311.01216/) (2023)
- [Stability and Generalization for Randomized Coordinate Descent](https://mlsanity.com/paper/2108.07414/) (2021)
- [Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets](https://mlsanity.com/paper/2111.14039/) (2021)
- [Convergence Rates of Training Deep Neural Networks via Alternating Minimization Methods](https://mlsanity.com/paper/2208.14318/) (2022)
- [Inertial Proximal Deep Learning Alternating Minimization for Efficient Neutral Network Training](https://mlsanity.com/paper/2102.00267/) (2021)
- [Stochastic Block-ADMM for Training Deep Networks](https://mlsanity.com/paper/2105.00339/) (2021)
- [Personalized On-Device E-health Analytics with Decentralized Block Coordinate Descent](https://mlsanity.com/paper/2112.09341/) (2021)
- [A Block Coordinate Descent Method for Nonsmooth Composite Optimization under Orthogonality Constraints](https://mlsanity.com/paper/2304.03641/) (2023)
- [Complexity of Block Coordinate Descent with Proximal Regularization and Applications to Wasserstein CP-dictionary Learning](https://mlsanity.com/paper/2306.02420/) (2023)
- [Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation](https://mlsanity.com/paper/2308.12053/) (2023)
- [Adaptive Pruning of Pretrained Transformer via Differential Inclusions](https://mlsanity.com/paper/2501.03289/) (2025)
- [A Convergent ADMM Framework for Efficient Neural Network Training](https://mlsanity.com/paper/2112.11619/) (2021)
- [Unsupervised Deep Learning Meets Chan-Vese Model](https://mlsanity.com/paper/2204.06951/) (2022)
- [On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guarantee](https://mlsanity.com/paper/2303.06815/) (2023)
- [A Proximal Algorithm for Network Slimming](https://mlsanity.com/paper/2307.00684/) (2023)
- [Component-based Sketching for Deep ReLU Nets](https://mlsanity.com/paper/2409.14174/) (2024)
- [CycleSL: Server-Client Cyclical Update Driven Scalable Split Learning](https://mlsanity.com/paper/2511.18611/) (2025)
- [Learning DNN networks using un-rectifying ReLU with compressed sensing application](https://mlsanity.com/paper/2101.06940/) (2021)
- [Communication Efficient Generalized Tensor Factorization for Decentralized Healthcare Networks](https://mlsanity.com/paper/2109.01718/) (2021)
- [On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vector](https://mlsanity.com/paper/2206.13646/) (2022)
- [ADMM Algorithms for Residual Network Training: Convergence Analysis and Parallel Implementation](https://mlsanity.com/paper/2310.15334/) (2023)

## Abstract (arXiv)

Deep learning has aroused extensive attention due to its great empirical success. The efficiency of the block coordinate descent (BCD) methods has been recently demonstrated in deep neural network (DNN) training. However, theoretical studies on their convergence properties are limited due to the highly nonconvex nature of DNN training. In this paper, we aim at providing a general methodology for provable convergence guarantees for this type of methods. In particular, for most of the commonly used DNN training models involving both two- and three-splitting schemes, we establish the global convergence to a critical point at a rate of cal O(1/k), where k is the number of iterations. The results extend to general loss functions which have Lipschitz continuous gradients and deep residual networks (ResNets). Our key development adds several new elements to the Kurdyka-Lojasiewicz inequality framework that enables us to carry out the global convergence analysis of BCD in the general scenario of deep learning.