mlsanityData to 7 Oct 2026

SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives

arXiv 1407.0202 ↗First posted 1 Jul 2014cs.LG math.OC stat.MLcited by 401

Not on the map. Why?

Abstract (arXiv)

In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for composite objectives where a proximal operator is used on the regulariser. Unlike SDCA, SAGA supports non-strongly convex problems directly, and is adaptive to any inherent strong convexity of the problem. We give experimental results showing the effectiveness of our method.

Key facts. Cited by 401 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jul 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 87; 2022: 93; 2023: 81; 2024: 53; 2025: 58; 2026: 29.

401 papers.

Sort:

By area:

  1. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  2. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  3. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  4. 2021Leveraging Sparse Linear Layers for Debuggable Deep Networkscited by 82
  5. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  6. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  7. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 63
  8. 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 62
  9. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 50
  10. 2023FFCV: Accelerating Training by Removing Data Bottleneckscited by 49
  11. 2021Stochastic Variance Reduction for Variational Inequality Methodscited by 46
  12. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 45
38 more
  1. 2023Stable Target Field for Reduced Variance Score Estimation in Diffusion Modelscited by 44
  2. 2022Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentationcited by 41
  3. 2023An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learningcited by 39
  4. 2021Secure Bilevel Asynchronous Vertical Federated Learning with Backward Updatingcited by 37
  5. 2022Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methodscited by 37
  6. 2023Stochastic Gradient Descent under Markovian Sampling Schemescited by 37
  7. 2021Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Featurescited by 36
  8. 2023Deep networks for system identification: a Surveycited by 35
  9. 2021Personalized Federated Learning: A Unified Framework and Universal Optimization Techniquescited by 34
  10. 2022SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compressioncited by 34
  11. 2021BiAdam: Fast Adaptive Bilevel Optimization Methodscited by 33
  12. 2022Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learningcited by 33
  13. 2021Unbalanced Optimal Transport through Non-negative Penalized Linear Regressioncited by 31
  14. 2023Optimal Algorithms for Stochastic Bilevel Optimization under Relaxed Smoothness Conditionscited by 31
  15. 2022From t-SNE to UMAP with contrastive learningcited by 28
  16. 2022Benchopt: Reproducible, efficient and collaborative optimization benchmarkscited by 27
  17. 2023Breaking the Lower Bound with (Little) Structure: Acceleration in Non-Convex Stochastic Optimization with Heavy-Tailed Noisecited by 27
  18. 2023Understanding Generalization of Federated Learning via Stability: Heterogeneity Matterscited by 27
  19. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  20. 2021An Adaptive Stochastic Sequential Quadratic Programming with Differentiable Exact Augmented Lagrangianscited by 25
  21. 2021Distributed Second Order Methods with Fast Rates and Compressed Communicationcited by 25
  22. 2022Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problemscited by 25
  23. 2023Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reductioncited by 23
  24. 2021Implicit differentiation for fast hyperparameter selection in non-smooth convex learningcited by 22
  25. 2021Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Controlcited by 22
  26. 2022Can 5th Generation Local Training Methods Support Client Sampling? Yes!cited by 22
  27. 2023Variance-reduced Clipping for Non-convex Optimizationcited by 22
  28. 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
  29. 2021Random Reshuffling with Variance Reduction: New Analysis and Better Ratescited by 19
  30. 2021Stochastic Polyak Stepsize with a Moving Targetcited by 19
  31. 2021FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learningcited by 19
  32. 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 19
  33. 2023Estimate-Then-Optimize versus Integrated-Estimation-Optimization versus Sample Average Approximation: A Stochastic Dominance Perspectivecited by 19
  34. 2021DeEPCA: Decentralized Exact PCA with Linear Convergence Ratecited by 18
  35. 2021Accelerated and instance-optimal policy evaluation with linear function approximationcited by 18
  36. 2022Walk for Learning: A Random Walk Approach for Federated Learning from Heterogeneous Datacited by 18
  37. 2022Faster federated optimization under second-order similaritycited by 18
  38. 2023Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraintscited by 18

50 most cited of 401 All 401 (JSON)