mlsanityData to 7 Oct 2026

Stochastic Variable Metric Proximal Gradient with variance reduction for non-convex composite optimization

arXiv 2301.00631 ↗First posted 2 Jan 2023cs.LG math.OCcited by 3

Not on the map. Why?

Abstract (arXiv)

This paper introduces a novel algorithm, the Perturbed Proximal Preconditioned SPIDER algorithm (3P-SPIDER), designed to solve finite sum non-convex composite optimization. It is a stochastic Variable Metric Forward-Backward algorithm, which allows approximate preconditioned forward operator and uses a variable metric proximity operator as the backward operator; it also proposes a mini-batch strategy with variance reduction to address the finite sum setting. We show that 3P-SPIDER extends some Stochastic preconditioned Gradient Descent-based algorithms and some Incremental Expectation Maximization algorithms to composite optimization and to the case the forward operator can not be computed in closed form. We also provide an explicit control of convergence in expectation of 3P-SPIDER, and study its complexity in order to satisfy the epsilon-approximate stationary condition. Our results are the first to combine the composite non-convex optimization setting, a variance reduction technique to tackle the finite sum setting by using a minibatch strategy and, to allow deterministic or random approximations of the preconditioned forward operator. Finally, through an application to inference in a logistic regression model with random effects, we numerically compare 3P-SPIDER to other stochastic forward-backward algorithms and discuss the role of some design parameters of 3P-SPIDER.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 28 of its 79 references that are in the map. First posted on arXiv 2 Jan 2023.

Builds on

28 of 79 references.

References · 28

  1. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  2. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  3. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  4. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 283
  5. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
  6. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  7. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 143
  8. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 142
  9. 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 116
  10. 2016Variance Reduction for Faster Non-Convex Optimizationcited by 72
  11. 2018Stochastic Nested Variance Reduction for Nonconvex Optimizationcited by 63
  12. 2017Natasha 2: Faster Non-Convex Optimization Than SGDcited by 49
16 more
  1. 2015A Linearly-Convergent Stochastic L-BFGS Algorithmcited by 39
  2. 2019ProxSARAH: An Efficient Algorithmic Framework for Stochastic Composite Nonconvex Optimizationcited by 37
  3. 2018A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimizationcited by 33
  4. 2019A Stochastic Composite Gradient Method with Incremental Variance Reductioncited by 26
  5. 2016Stochastic Block BFGS: Squeezing More Curvature out of Datacited by 24
  6. 2020Adaptivity of Stochastic Gradient Methods for Nonconvex Optimizationcited by 16
  7. 2019On the Global Convergence of (Fast) Incremental Expectation Maximization Methodscited by 12
  8. 2018On Quasi-Newton Forward--Backward Splitting: Proximal Calculus and Convergencecited by 8
  9. 2020Faster Stochastic Quasi-Newton Methodscited by 8
  10. 2020Geom-SPIDER-EM: Faster Variance Reduced Stochastic Expectation Maximization for Nonconvex Finite-Sum Optimizationcited by 7
  11. 2020Fast Incremental Expectation Maximization for finite-sum optimization: nonasymptotic convergencecited by 7
  12. 2019Stochastic Proximal Methods for Non-Smooth Non-Convex Constrained Sparse Optimizationcited by 4
  13. 2019Variable Metric Proximal Gradient Method with Diagonal Barzilai-Borwein Stepsizecited by 4
  14. 2021The perturbed prox-preconditioned spider algorithm: non-asymptotic convergence boundscited by 4
  15. 2019Variable Metric Forward-Backward Algorithm for Composite Minimization Problemscited by 3
  16. 2020A Stochastic Path-Integrated Differential EstimatoR Expectation Maximization Algorithmcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 0; 2026: 2.

3 papers.

  1. 2023Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learningcited by 10
  2. 2026Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Expertscited by 0
  3. 2026Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Modelscited by 0