# Stochastic Block-ADMM for Training Deep Networks

arXiv: https://arxiv.org/abs/2105.00339 · HTML: https://mlsanity.com/paper/2105.00339/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 1 May 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Global Convergence of ADMM in Nonconvex Nonsmooth Optimization](https://mlsanity.com/paper/1511.06324/) (2015)
- [Deep Feature Factorization For Concept Discovery](https://mlsanity.com/paper/1806.10206/) (2018)
- [Training Neural Networks Without Gradients: A Scalable ADMM Approach](https://mlsanity.com/paper/1605.02026/) (2016)
- [Artificial Intelligence and Statistics](https://mlsanity.com/paper/1712.03779/) (2017)
- [Global Convergence of Block Coordinate Descent in Deep Learning](https://mlsanity.com/paper/1803.00225/) (2018)
- [Exploring Disentangled Feature Representation Beyond Face Identification](https://mlsanity.com/paper/1804.03487/) (2018)
- [Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks](https://mlsanity.com/paper/1711.07354/) (2017)
- [PANDA: Pose Aligned Networks for Deep Attribute Modeling](https://mlsanity.com/paper/1311.5591/) (2013)
- [Beyond Backprop: Online Alternating Minimization with Auxiliary Variables](https://mlsanity.com/paper/1806.09077/) (2018)
- [Block stochastic gradient iteration for convex and nonconvex optimization](https://mlsanity.com/paper/1408.2597/) (2014)
- [Lifted Neural Networks](https://mlsanity.com/paper/1805.01532/) (2018)
- [ADMM for Efficient Deep Learning with Global Convergence](https://mlsanity.com/paper/1905.13611/) (2019)
- [Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training](https://mlsanity.com/paper/1811.08039/) (2018)
- [Mini-Batch Stochastic ADMMs for Nonconvex Nonsmooth Optimization](https://mlsanity.com/paper/1802.03284/) (2018)

## Built on by (most cited first, 2 of 2)

- [Cyclic Block Coordinate Descent With Variance Reduction for Composite Nonconvex Optimization](https://mlsanity.com/paper/2212.05088/) (2022)
- [A Convergent ADMM Framework for Efficient Neural Network Training](https://mlsanity.com/paper/2112.11619/) (2021)

## Abstract (arXiv)

In this paper, we propose Stochastic Block-ADMM as an approach to train deep neural networks in batch and online settings. Our method works by splitting neural networks into an arbitrary number of blocks and utilizes auxiliary variables to connect these blocks while optimizing with stochastic gradient descent. This allows training deep networks with non-differentiable constraints where conventional backpropagation is not applicable. An application of this is supervised feature disentangling, where our proposed DeepFacto inserts a non-negative matrix factorization (NMF) layer into the network. Since backpropagation only needs to be performed within each block, our approach alleviates vanishing gradients and provides potentials for parallelization. We prove the convergence of our proposed method and justify its capabilities through experiments in supervised and weakly-supervised settings.