mlsanityData to 7 Oct 2026

Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey

arXiv 1507.01030 ↗First posted 3 Jul 2015eess.SY cs.DS cs.NAcited by 44

Not on the map. Why?

Abstract (arXiv)

We survey incremental methods for minimizing a sum sumi=1mfi(x) consisting of a large number of convex component functions fi. Our methods consist of iterations applied to single components, and have proved very effective in practice. We introduce a unified algorithmic framework for a variety of such methods, some involving gradient and subgradient iterations, which are known, and some involving combinations of subgradient and proximal methods, which are new and offer greater flexibility in exploiting the special structure of fi. We provide an analysis of the convergence and rate of convergence properties of these methods, including the advantages offered by randomization in the selection of components. We also survey applications in inference/machine learning, signal processing, and large-scale and distributed optimization.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jul 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 8; 2023: 9; 2024: 12; 2025: 5; 2026: 4.

44 papers.

Sort:

By area:

  1. 2021Proximal and Federated Random Reshufflingcited by 33
  2. 2022Particle-based Variational Inference with Preconditioned Functional Gradient Flowcited by 20
  3. 2022GraB: Finding Provably Better Data Permutations than Random Reshufflingcited by 18
  4. 2021Permutation-Based SGD: Is Random Optimal?cited by 16
  5. 2022Sampling without Replacement Leads to Faster Rates in Finite-Sum Minimax Optimizationcited by 10
  6. 2024Last Iterate Convergence of Incremental Methods and Applications in Continual Learningcited by 10
  7. 2024On the Last-Iterate Convergence of Shuffling Gradient Methodscited by 9
  8. 2021Faster Deep Reinforcement Learning with Slower Online Networkcited by 7
  9. 2023Coordinating Distributed Example Orders for Provably Accelerated Trainingcited by 7
  10. 2023On Convergence of Incremental Gradient for Non-Convex Smooth Functionscited by 7
  11. 2022Federated Learning with Server Learning: Enhancing Performance for Non-IID Datacited by 6
  12. 2023Linear Speedup of Incremental Aggregated Gradient Methods on Streaming Datacited by 5
32 more
  1. 2023Personalized Federated Learning with Attention-based Client Selectioncited by 5
  2. 2024The Entrapment Problem in Random Walk Decentralized Learningcited by 4
  3. 2022FedMCSA: Personalized Federated Learning via Model Components Self-Attentioncited by 3
  4. 2023Convergence of stochastic gradient descent under a local Lojasiewicz condition for deep neural networkscited by 3
  5. 2023Generalized Low-Rank Update: Model Parameter Bounds for Low-Rank Training Data Modificationscited by 3
  6. 2024The Diversity Bonus: Learning from Dissimilar Distributed Clients in Personalized Federated Learningcited by 3
  7. 2021Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Serverscited by 2
  8. 2021Distributed stochastic proximal algorithm with random reshuffling for non-smooth finite-sum optimizationcited by 2
  9. 2023DualFL: A Duality-based Federated Learning Algorithm with Communication Acceleration in the General Convex Regimecited by 2
  10. 2024Understanding the Training Speedup from Sampling with Approximate Lossescited by 2
  11. 2024Parallel Momentum Methods Under Biased Gradient Estimationscited by 2
  12. 2024On the Convergence of Federated Learning Algorithms without Data Similaritycited by 2
  13. 2024Robust Low-rank Tensor Train Recoverycited by 2
  14. 2025Single-shot Star-convex Polygon-based Instance Segmentation for Spatially-correlated Biomedical Objectscited by 2
  15. 2022Deep Neural Network Based Accelerated Failure Time Models using Rank Losscited by 1
  16. 2022Proximal Mean Field Learning in Shallow Neural Networkscited by 1
  17. 2024FedSR: A Semi-Decentralized Federated Learning Algorithm for Non-IIDness in IoT Systemcited by 1
  18. 2024Artificial Intelligence for the Internal Democracy of Political Partiescited by 1
  19. 2024A simple and improved algorithm for noisy, convex, zeroth-order optimisationcited by 1
  20. 2025Online Gradient Boosting Decision Tree: In-Place Updates for Efficient Adding/Deleting Datacited by 1
  21. 2025Accelerated Gradient Methods with Biased Gradient Estimates: Risk Sensitivity, High-Probability Guarantees, and Large Deviation Boundscited by 1
  22. 2021Adaptive Stochastic ADMM for Decentralized Reinforcement Learning in Edge Industrial IoTcited by 0
  23. 2022Convergence of ease-controlled Random Reshuffling gradient Algorithms under Lipschitz smoothnesscited by 0
  24. 2023Ordering for Non-Replacement SGDcited by 0
  25. 2023Physical-Layer Semantic-Aware Network for Zero-Shot Wireless Sensingcited by 0
  26. 2024Stochastic optimization with arbitrary recurrent data samplingcited by 0
  27. 2025Mean-square and sublinear convergence of a stochastic proximal point algorithm in metric spaces of nonpositive curvaturecited by 0
  28. 2025First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptionscited by 0
  29. 2026A Short and Unified Convergence Analysis of the SAG, SAGA, and IAG Algorithmscited by 0
  30. 2026Shuffling the Data, Stretching the Step-size: Sharper Bias in constant step-size SGDcited by 0
  31. 2026Decentralized Learning via Random Walk with Jumpscited by 0
  32. 2026Heterogeneity-Aware Personalized Federated Learning for Industrial Predictive Analyticscited by 0