# Stochastic Variance-Reduced ADMM

arXiv: https://arxiv.org/abs/1604.07070 · HTML: https://mlsanity.com/paper/1604.07070/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2016.

## Built on by (most cited first, 5 of 5)

- [Cyclic Block Coordinate Descent With Variance Reduction for Composite Nonconvex Optimization](https://mlsanity.com/paper/2212.05088/) (2022)
- [Privacy Amplification by Iteration for ADMM with (Strongly) Convex Objective Functions](https://mlsanity.com/paper/2312.08685/) (2023)
- [Robust Regression via Model Based Methods](https://mlsanity.com/paper/2106.10759/) (2021)
- [Provable Constrained Stochastic Convex Optimization with XOR-Projected Gradient Descent](https://mlsanity.com/paper/2203.11829/) (2022)
- [A General Continuous-Time Formulation of Stochastic ADMM and Its Variants](https://mlsanity.com/paper/2404.14358/) (2024)

## Abstract (arXiv)

The alternating direction method of multipliers (ADMM) is a powerful optimization solver in machine learning. Recently, stochastic ADMM has been integrated with variance reduction methods for stochastic gradient, leading to SAG-ADMM and SDCA-ADMM that have fast convergence rates and low iteration complexities. However, their space requirements can still be high. In this paper, we propose an integration of ADMM with the method of stochastic variance reduced gradient (SVRG). Unlike another recent integration attempt called SCAS-ADMM, the proposed algorithm retains the fast convergence benefits of SAG-ADMM and SDCA-ADMM, but is more advantageous in that its storage requirement is very low, even independent of the sample size n. We also extend the proposed method for nonconvex problems, and obtain a convergence rate of O(1/T). Experimental results demonstrate that it is as fast as SAG-ADMM and SDCA-ADMM, much faster than SCAS-ADMM, and can be used on much bigger data sets.