mlsanityData to 7 Oct 2026

An Adaptive Stochastic Sequential Quadratic Programming with Differentiable Exact Augmented Lagrangians

arXiv 2102.05320 ↗First posted 10 Feb 2021math.OC cs.NA math.NAcited by 25

Not on the map. Why?

Abstract (arXiv)

We consider solving nonlinear optimization problems with a stochastic objective and deterministic equality constraints. We assume for the objective that its evaluation, gradient, and Hessian are inaccessible, while one can compute their stochastic estimates by, for example, subsampling. We propose a stochastic algorithm based on sequential quadratic programming (SQP) that uses a differentiable exact augmented Lagrangian as the merit function. To motivate our algorithm design, we first revisit and simplify an old SQP method citepLucidi1990Recursive developed for solving deterministic problems, which serves as the skeleton of our stochastic algorithm. Based on the simplified deterministic algorithm, we then propose a non-adaptive SQP for dealing with stochastic objective, where the gradient and Hessian are replaced by stochastic estimates but the stepsizes are deterministic and prespecified. Finally, we incorporate a recent stochastic line search procedure citepPaquette2020Stochastic into the non-adaptive stochastic SQP to adaptively select the random stepsizes, which leads to an adaptive stochastic SQP. The global "almost sure" convergence for both non-adaptive and adaptive SQP methods is established. Numerical experiments on nonlinear problems in CUTEst test set demonstrate the superiority of the adaptive algorithm.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 61 of its 229 references that are in the map. First posted on arXiv 10 Feb 2021.

Builds on

61 of 229 references.

References · 61

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  4. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  5. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  6. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  7. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  8. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  9. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  10. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  11. 2017Stochastic Cubic Regularization for Fast Nonconvex Optimizationcited by 55
  12. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 55
38 more
  1. 2015Equilibrated adaptive learning rates for non-convex optimizationcited by 42
  2. 2016Efficient Distributed Learning with Sparsitycited by 38
  3. 2016Exact and Inexact Subsampled Newton Methods for Optimizationcited by 38
  4. 2016Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimizationcited by 37
  5. 2014Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squarescited by 34
  6. 2016Efficient Second Order Online Learning by Sketchingcited by 33
  7. 2015Convergence rates of sub-sampled Newton methodscited by 32
  8. 2020Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approachcited by 31
  9. 2020Sequential Quadratic Optimization for Nonlinear Equality Constrained Stochastic Optimizationcited by 30
  10. 2017Adaptive Sampling Strategies for Stochastic Optimizationcited by 29
  11. 2015Global convergence rate analysis of unconstrained optimization methods based on probabilistic modelscited by 27
  12. 2015Stochastic Optimization Using a Trust-Region Method and Random Modelscited by 25
  13. 2014Coauthorship and Citation Networks for Statisticianscited by 22
  14. 2017An Investigation of Newton-Sketch and Subsampled Newton Methodscited by 18
  15. 2017Sketched Ridge Regression: Optimization Perspective, Statistical Perspective, and Model Averagingcited by 13
  16. 2015Distributed Multitask Learningcited by 11
  17. 2013Convergence of trust-region methods based on probabilistic modelscited by 10
  18. 2017A Stochastic Trust Region Algorithm Based on Careful Step Normalizationcited by 10
  19. 2019Direct Estimation of Differential Functional Graphical Modelscited by 8
  20. 2019A Fully Stochastic Second-Order Trust Region Methodcited by 8
  21. 2019Two-sample inference for high-dimensional Markov networkscited by 7
  22. 2019Estimating Differential Latent Variable Graphical Models with Applications to Brain Connectivitycited by 7
  23. 2020Convergence Analysis of Accelerated Stochastic Gradient Descent under the Growth Conditioncited by 7
  24. 2016Sketching Meets Random Projection in the Dual: A Provable Recovery Algorithm for Big and High-dimensional Datacited by 6
  25. 2018Distributed Stochastic Multi-Task Learning with Graph Regularizationcited by 6
  26. 2018High-dimensional Varying Index Coefficient Models via Stein's Identitycited by 6
  27. 2019Partially Linear Additive Gaussian Graphical Modelscited by 6
  28. 2014Optimal variable selection in multi-group sparse discriminant analysiscited by 5
  29. 2015ROCKET: Robust Confidence Intervals via Kendall's Tau for Transelliptical Graphical Modelscited by 5
  30. 2015Kernel Meets Sieve: Post-Regularization Confidence Bands for Sparse Additive Modelcited by 5
  31. 2016Distributed Multi-Task Learning with Shared Representationcited by 5
  32. 2018Recovery of simultaneous low rank and two-way sparse coefficient matrices, a nonconvex approachcited by 5
  33. 2019Simultaneous Inference for Pairwise Graphical Models with Generalized Score Matchingcited by 5
  34. 2020Statistical Inference for Networks of High-Dimensional Point Processescited by 5
  35. 2013Optimal Feature Selection in High-Dimensional Discriminant Analysiscited by 4
  36. 2014A General Framework for Robust Testing and Confidence Regions in High-Dimensional Quantile Regressioncited by 4
  37. 2018Joint Nonparametric Precision Matrix Estimation with Confoundingcited by 4
  38. 2020LSOS: Line-search Second-Order Stochastic optimization methods for nonconvex finite sumscited by 4

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 3; 2023: 5; 2024: 5; 2025: 6; 2026: 4.

25 papers.

Sort:

  1. 2021A Stochastic Sequential Quadratic Optimization Algorithm for Nonlinear Equality Constrained Optimization with Rank-Deficient Jacobianscited by 21
  2. 2022Fully Stochastic Trust-Region Sequential Quadratic Programming for Equality-Constrained Optimization Problemscited by 21
  3. 2021Inequality Constrained Stochastic Nonlinear Optimization via Active-Set Sequential Quadratic Programmingcited by 16
  4. 2022Hessian Averaging in Stochastic Newton Methods Achieves Superlinear Convergencecited by 14
  5. 2023Oracle Complexity of Single-Loop Switching Subgradient Methods for Non-Smooth Weakly Convex Functional Constrained Optimizationcited by 11
  6. 2024Variance-reduced first-order methods for deterministically constrained stochastic nonconvex optimization with strong convergence guaranteescited by 11
  7. 2022Statistical Inference of Constrained Stochastic Optimization via Sketched Sequential Quadratic Programmingcited by 10
  8. 2023A Sequential Quadratic Programming Method with High Probability Complexity Bounds for Nonlinear Equality Constrained Stochastic Optimizationcited by 8
  9. 2023A Stochastic-Gradient-based Interior-Point Algorithm for Solving Smooth Bound-Constrained Optimization Problemscited by 8
  10. 2023Constrained Optimization via Exact Augmented Lagrangian and Randomized Iterative Sketchingcited by 5
  11. 2023Almost-sure convergence of iterates and multipliers in stochastic sequential quadratic optimizationcited by 5
  12. 2024Trust-Region Sequential Quadratic Programming for Stochastic Optimization with Random Modelscited by 5
13 more
  1. 2025Single-loop Algorithms for Stochastic Non-convex Optimization with Weakly-Convex Constraintscited by 5
  2. 2024Physics-Informed Neural Networks with Trust-Region Sequential Quadratic Programmingcited by 4
  3. 2025High Probability Complexity Bounds of Trust-Region Stochastic Sequential Quadratic Programming with Heavy-Tailed Noisecited by 4
  4. 2024Single-Loop Deterministic and Stochastic Interior-Point Algorithms for Nonlinearly Constrained Optimizationcited by 1
  5. 2024Stochastic interior-point methods for smooth conic optimization with applicationscited by 1
  6. 2026A Trust-Region Interior-Point Stochastic Sequential Quadratic Programming Methodcited by 1
  7. 2025Mirror Descent Linearized Augmented Lagrangian Methods for Nonconvex Constrained Stochastic Zeroth-Order Optimizationcited by 0
  8. 2025Online Statistical Inference of Constrained Stochastic Optimization via Random Scalingcited by 0
  9. 2025Benchmarking Stochastic Approximation Algorithms for Fairness-Constrained Training of Deep Neural Networkscited by 0
  10. 2025humancompatible.train: Implementing Optimization Algorithms for Stochastically-Constrained Stochastic Optimization Problemscited by 0
  11. 2026TRSVR: An Adaptive Stochastic Trust-Region Method with Variance Reductioncited by 0
  12. 2026MoSSP: A Momentum-Based Single-Loop Stochastic Penalty Method for Nonconvex Constrained DC-Regularized Optimizationcited by 0
  13. 2026TR-SSQP: A Trust-Region Method for Constrained Stochastic Optimization under Heavy-Tailed Noisecited by 0