mlsanityData to 14 Oct 2026

Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks

arXiv 1711.07354 ↗First posted 20 Nov 2017stat.ML cs.CV cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

By lifting the ReLU function into a higher dimensional space, we develop a smooth multi-convex formulation for training feed-forward deep neural networks (DNNs). This allows us to develop a block coordinate descent (BCD) training algorithm consisting of a sequence of numerically well-behaved convex optimizations. Using ideas from proximal point methods in convex analysis, we prove that this BCD algorithm will converge globally to a stationary point with R-linear convergence rate of order one. In experiments with the MNIST database, DNNs trained with this BCD algorithm consistently yielded better test-set error rates than identical DNN architectures trained via all the stochastic gradient descent (SGD) variants in the Caffe toolbox.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 4; 2023: 4; 2024: 2; 2025: 4; 2026: 1.

26 papers.

Sort:

  1. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 78
  2. 2021On Training Implicit Modelscited by 75
  3. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  4. 2021LocoProp: Enhancing BackProp via Local Loss Optimizationcited by 15
  5. 2023Dual Propagation: Accelerating Contrastive Hebbian Learning with Dyadic Neuronscited by 14
  6. 2022Deep Incubation: Training Large Models by Divide-and-Conqueringcited by 13
  7. 20220/1 Deep Neural Networks via Block Coordinate Descentcited by 10
  8. 2021Modeling Design and Control Problems Involving Neural Network Surrogatescited by 9
  9. 2021Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methodscited by 5
  10. 2021Bilevel Programs Meet Deep Learning: A Unifying View on Inference Learning Methodscited by 5
  11. 2022Convergence Rates of Training Deep Neural Networks via Alternating Minimization Methodscited by 4
  12. 2024Towards training digitally-tied analog blocks via hybrid gradient computationcited by 4
14 more
  1. 2021Inertial Proximal Deep Learning Alternating Minimization for Efficient Neutral Network Trainingcited by 3
  2. 2024A primal-dual algorithm for image reconstruction with input-convex neural network regularizerscited by 3
  3. 2021Stochastic Block-ADMM for Training Deep Networkscited by 2
  4. 2021Personalized On-Device E-health Analytics with Decentralized Block Coordinate Descentcited by 2
  5. 2023Complexity of Block Coordinate Descent with Proximal Regularization and Applications to Wasserstein CP-dictionary Learningcited by 2
  6. 2023On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guaranteecited by 1
  7. 2025A Triple-Inertial Accelerated Alternating Optimization Method for Deep Learning Trainingcited by 1
  8. 2025CycleSL: Server-Client Cyclical Update Driven Scalable Split Learningcited by 1
  9. 2021Learning DNN networks using un-rectifying ReLU with compressed sensing applicationcited by 0
  10. 2021Training Deep Neural Networks via Branch-and-Boundcited by 0
  11. 2021Transformer-Encoder-GRU (T-E-GRU) for Chinese Sentiment Analysis on Chinese Comment Textcited by 0
  12. 2025Exploiting Block Coordinate Descent for Cost-Effective LLM Model Trainingcited by 0
  13. 2025Block Coordinate Descent for Neural Networks Provably Finds Global Minimacited by 0
  14. 2026A block-coordinate descent framework for non-convex composite optimization. Application to sparse precision matrix estimationcited by 0