# A General Continuous-Time Formulation of Stochastic ADMM and Its Variants

arXiv: https://arxiv.org/abs/2404.14358 · HTML: https://mlsanity.com/paper/2404.14358/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 81 references that are in the map.
- First posted on arXiv 22 Apr 2024.

## Builds on (most cited first, 30 of 32)

- [A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights](https://mlsanity.com/paper/1503.01243/) (2015)
- [Stochastic modified equations and adaptive stochastic gradient algorithms](https://mlsanity.com/paper/1511.06251/) (2015)
- [Underdamped Langevin MCMC: A non-asymptotic analysis](https://mlsanity.com/paper/1707.03663/) (2017)
- [Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations](https://mlsanity.com/paper/1811.01558/) (2018)
- [A Variational Perspective on Accelerated Methods in Optimization](https://mlsanity.com/paper/1603.04245/) (2016)
- [A Hitting Time Analysis of Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/1702.05575/) (2017)
- [Understanding the Acceleration Phenomenon via High-Resolution Differential Equations](https://mlsanity.com/paper/1810.08907/) (2018)
- [Neural Proximal Gradient Descent for Compressive Imaging](https://mlsanity.com/paper/1806.03963/) (2018)
- [A Three-Operator Splitting Scheme and its Optimization Applications](https://mlsanity.com/paper/1504.01032/) (2015)
- [On Learning Rates and Schr"odinger Operators](https://mlsanity.com/paper/2004.06977/) (2020)
- [Global Convergence of Stochastic Gradient Hamiltonian Monte Carlo for Non-Convex Stochastic Optimization: Non-Asymptotic Performance Bounds and Momentum-Based Acceleration](https://mlsanity.com/paper/1809.04618/) (2018)
- [A Lyapunov Analysis of Momentum Methods in Optimization](https://mlsanity.com/paper/1611.02635/) (2016)
- [Acceleration via Symplectic Discretization of High-Resolution Differential Equations](https://mlsanity.com/paper/1902.03694/) (2019)
- [On dissipative symplectic integration with applications to gradient-based optimization](https://mlsanity.com/paper/2004.06840/) (2020)
- [A General Analysis of the Convergence of ADMM](https://mlsanity.com/paper/1502.02009/) (2015)
- [Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximation](https://mlsanity.com/paper/1902.00635/) (2019)
- [An Accelerated Linearized Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1401.6607/) (2014)
- [Alternating proximal gradient method for sparse nonnegative Tucker decomposition](https://mlsanity.com/paper/1302.2559/) (2013)
- [Sparse Recovery via Differential Inclusions](https://mlsanity.com/paper/1406.7728/) (2014)
- [Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimization](https://mlsanity.com/paper/2008.01296/) (2020)
- [ADMM and Accelerated ADMM as Continuous Dynamical Systems](https://mlsanity.com/paper/1805.06579/) (2018)
- [Online ICA: Understanding Global Dynamics of Nonconvex Optimization via Diffusion Processes](https://mlsanity.com/paper/1808.09642/) (2018)
- [Trajectory of Alternating Direction Method of Multipliers and Adaptive Acceleration](https://mlsanity.com/paper/1906.10114/) (2019)
- [Stochastic Variance-Reduced ADMM](https://mlsanity.com/paper/1604.07070/) (2016)
- [On the diffusion approximation of nonconvex stochastic gradient descent](https://mlsanity.com/paper/1705.07562/) (2017)
- [Accelerated Variance Reduced Stochastic ADMM](https://mlsanity.com/paper/1707.03190/) (2017)
- [A Nonsmooth Dynamical Systems Perspective on Accelerated Extensions of ADMM](https://mlsanity.com/paper/1808.04048/) (2018)
- [Degrees of Freedom Analysis of Unrolled Neural Networks](https://mlsanity.com/paper/1906.03742/) (2019)
- [Fast Stochastic Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1308.3558/) (2013)
- [Diffusion Approximations for Online Principal Component Estimation and Global Convergence](https://mlsanity.com/paper/1808.09645/) (2018)

## Abstract (arXiv)

Stochastic versions of the alternating direction method of multiplier (ADMM) and its variants play a key role in many modern large-scale machine learning problems. In this work, we introduce a unified algorithmic framework called generalized stochastic ADMM and investigate their continuous-time analysis. The generalized framework widely includes many stochastic ADMM variants such as standard, linearized and gradient-based ADMM. Our continuous-time analysis provides us with new insights into stochastic ADMM and variants, and we rigorously prove that under some proper scaling, the trajectory of stochastic ADMM weakly converges to the solution of a stochastic differential equation with small noise. Our analysis also provides a theoretical explanation of why the relaxation parameter should be chosen between 0 and 2.