mlsanityData to 7 Oct 2026

Global Convergence of Block Coordinate Descent in Deep Learning

arXiv 1803.00225 ↗First posted 1 Mar 2018math.OC cs.LG stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

Deep learning has aroused extensive attention due to its great empirical success. The efficiency of the block coordinate descent (BCD) methods has been recently demonstrated in deep neural network (DNN) training. However, theoretical studies on their convergence properties are limited due to the highly nonconvex nature of DNN training. In this paper, we aim at providing a general methodology for provable convergence guarantees for this type of methods. In particular, for most of the commonly used DNN training models involving both two- and three-splitting schemes, we establish the global convergence to a critical point at a rate of cal O(1/k), where k is the number of iterations. The results extend to general loss functions which have Lipschitz continuous gradients and deep residual networks (ResNets). Our key development adds several new elements to the Kurdyka-Lojasiewicz inequality framework that enables us to carry out the global convergence analysis of BCD in the general scenario of deep learning.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 5; 2023: 7; 2024: 4; 2025: 7; 2026: 3.

38 papers.

Sort:

  1. 2025On the Convergence of Muon and Beyondcited by 28
  2. 2021LocoProp: Enhancing BackProp via Local Loss Optimizationcited by 15
  3. 2021Nonlinear Transform Induced Tensor Nuclear Norm for Tensor Completioncited by 13
  4. 20220/1 Deep Neural Networks via Block Coordinate Descentcited by 10
  5. 2024BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blockscited by 9
  6. 2026FedBCD:Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learningcited by 7
  7. 2021Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methodscited by 5
  8. 2021Coordinate descent on the orthogonal group for recurrent neural network trainingcited by 5
  9. 2022An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsitycited by 5
  10. 2023Convergent plug-and-play with proximal denoiser and unconstrained regularization parametercited by 5
  11. 2021Stability and Generalization for Randomized Coordinate Descentcited by 4
  12. 2021Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Netscited by 4
26 more
  1. 2022Convergence Rates of Training Deep Neural Networks via Alternating Minimization Methodscited by 4
  2. 2021Inertial Proximal Deep Learning Alternating Minimization for Efficient Neutral Network Trainingcited by 3
  3. 2021Stochastic Block-ADMM for Training Deep Networkscited by 2
  4. 2021Personalized On-Device E-health Analytics with Decentralized Block Coordinate Descentcited by 2
  5. 2023A Block Coordinate Descent Method for Nonsmooth Composite Optimization under Orthogonality Constraintscited by 2
  6. 2023Complexity of Block Coordinate Descent with Proximal Regularization and Applications to Wasserstein CP-dictionary Learningcited by 2
  7. 2023Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagationcited by 2
  8. 2025Adaptive Pruning of Pretrained Transformer via Differential Inclusionscited by 2
  9. 2021A Convergent ADMM Framework for Efficient Neural Network Trainingcited by 1
  10. 2022Unsupervised Deep Learning Meets Chan-Vese Modelcited by 1
  11. 2023On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guaranteecited by 1
  12. 2023A Proximal Algorithm for Network Slimmingcited by 1
  13. 2024Component-based Sketching for Deep ReLU Netscited by 1
  14. 2025CycleSL: Server-Client Cyclical Update Driven Scalable Split Learningcited by 1
  15. 2021Learning DNN networks using un-rectifying ReLU with compressed sensing applicationcited by 0
  16. 2021Communication Efficient Generalized Tensor Factorization for Decentralized Healthcare Networkscited by 0
  17. 2022On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vectorcited by 0
  18. 2023ADMM Algorithms for Residual Network Training: Convergence Analysis and Parallel Implementationcited by 0
  19. 2024Langevin Dynamics: A Unified Perspective on Optimization via Lyapunov Potentialscited by 0
  20. 2024On ADMM in Heterogeneous Federated Learning: Personalization, Robustness, and Fairnesscited by 0
  21. 2025Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Conditioncited by 0
  22. 2025Exploiting Block Coordinate Descent for Cost-Effective LLM Model Trainingcited by 0
  23. 2025Neural Network Training via Stochastic Alternating Minimization with Trainable Step Sizescited by 0
  24. 2025Block Coordinate Descent for Neural Networks Provably Finds Global Minimacited by 0
  25. 2026A block-coordinate descent framework for non-convex composite optimization. Application to sparse precision matrix estimationcited by 0
  26. 2026AltTS: A Dual-Path Framework with Alternating Optimization for Multivariate Time Series Forecastingcited by 0