A General Continuous-Time Formulation of Stochastic ADMM and Its Variants
Not on the map. Why?
Abstract (arXiv)
Stochastic versions of the alternating direction method of multiplier (ADMM) and its variants play a key role in many modern large-scale machine learning problems. In this work, we introduce a unified algorithmic framework called generalized stochastic ADMM and investigate their continuous-time analysis. The generalized framework widely includes many stochastic ADMM variants such as standard, linearized and gradient-based ADMM. Our continuous-time analysis provides us with new insights into stochastic ADMM and variants, and we rigorously prove that under some proper scaling, the trajectory of stochastic ADMM weakly converges to the solution of a stochastic differential equation with small noise. Our analysis also provides a theoretical explanation of why the relaxation parameter should be chosen between 0 and 2.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 81 references that are in the map. First posted on arXiv 22 Apr 2024.
Builds on
32 of 81 references.
References · 32
- 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
- 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
- 2017Underdamped Langevin MCMC: A non-asymptotic analysiscited by 133
- 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
- 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 110
- 2017A Hitting Time Analysis of Stochastic Gradient Langevin Dynamicscited by 71
- 2018Understanding the Acceleration Phenomenon via High-Resolution Differential Equationscited by 69
- 2018Neural Proximal Gradient Descent for Compressive Imagingcited by 46
- 2015A Three-Operator Splitting Scheme and its Optimization Applicationscited by 34
- 2020On Learning Rates and Schr"odinger Operatorscited by 33
- 2018Global Convergence of Stochastic Gradient Hamiltonian Monte Carlo for Non-Convex Stochastic Optimization: Non-Asymptotic Performance Bounds and Momentum-Based Accelerationcited by 31
- 2016A Lyapunov Analysis of Momentum Methods in Optimizationcited by 28
20 more
- 2019Acceleration via Symplectic Discretization of High-Resolution Differential Equationscited by 27
- 2020On dissipative symplectic integration with applications to gradient-based optimizationcited by 23
- 2015A General Analysis of the Convergence of ADMMcited by 22
- 2019Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximationcited by 15
- 2014An Accelerated Linearized Alternating Direction Method of Multiplierscited by 14
- 2013Alternating proximal gradient method for sparse nonnegative Tucker decompositioncited by 11
- 2014Sparse Recovery via Differential Inclusionscited by 11
- 2020Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimizationcited by 10
- 2018ADMM and Accelerated ADMM as Continuous Dynamical Systemscited by 6
- 2018Online ICA: Understanding Global Dynamics of Nonconvex Optimization via Diffusion Processescited by 6
- 2019Trajectory of Alternating Direction Method of Multipliers and Adaptive Accelerationcited by 6
- 2016Stochastic Variance-Reduced ADMMcited by 5
- 2017On the diffusion approximation of nonconvex stochastic gradient descentcited by 5
- 2017Accelerated Variance Reduced Stochastic ADMMcited by 5
- 2018A Nonsmooth Dynamical Systems Perspective on Accelerated Extensions of ADMMcited by 4
- 2019Degrees of Freedom Analysis of Unrolled Neural Networkscited by 4
- 2013Fast Stochastic Alternating Direction Method of Multiplierscited by 3
- 2018Diffusion Approximations for Online Principal Component Estimation and Global Convergencecited by 3
- 2013An Extragradient-Based Alternating Direction Method for Convex Minimizationcited by 2
- 2017Fast Stochastic Variance Reduced ADMM for Stochastic Composition Optimizationcited by 2
Built on by
Not cited yet.