mlsanityData to 7 Oct 2026

Optimization Methods for Large-Scale Machine Learning

arXiv 1606.04838 ↗First posted 15 Jun 2016stat.ML cs.LG math.OCcited by 1,140

Not on the map. Why?

Abstract (arXiv)

This paper provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications. Through case studies on text classification and the training of deep neural networks, we discuss how optimization problems arise in machine learning and what makes them challenging. A major theme of our study is that large-scale machine learning represents a distinctive setting in which the stochastic gradient (SG) method has traditionally played a central role while conventional gradient-based nonlinear optimization techniques typically falter. Based on this viewpoint, we present a comprehensive theory of a straightforward, yet versatile SG algorithm, discuss its practical behavior, and highlight opportunities for designing algorithms with improved performance. This leads to a discussion about the next generation of optimization methods for large-scale machine learning, including an investigation of two main streams of research on techniques that diminish noise in the stochastic directions and methods that make use of second-order derivative approximations.

Key facts. Cited by 1,140 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 218; 2022: 183; 2023: 179; 2024: 196; 2025: 167; 2026: 197.

1,140 papers.

Sort:

By area:

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  2. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  3. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  4. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  5. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  6. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  7. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  8. 2022Federated Learning with Partial Model Personalizationcited by 114
  9. 2021An Introduction to Deep Generative Modelingcited by 104
  10. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  11. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  12. 2021Bayesian imaging using Plug & Play priors: when Langevin meets Tweediecited by 96
38 more
  1. 2021The Bayesian Learning Rulecited by 95
  2. 2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
  3. 2024AI and Memory Wallcited by 89
  4. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  5. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  6. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  7. 2022When Do Flat Minima Optimizers Work?cited by 78
  8. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  9. 2021A general sample complexity analysis of vanilla policy gradientcited by 74
  10. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  11. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  12. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  13. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  14. 2021DAS-PINNs: A deep adaptive sampling method for solving high-dimensional partial differential equationscited by 66
  15. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 66
  16. 2023Fixing by Mixing: A Recipe for Optimal Byzantine ML under Heterogeneitycited by 64
  17. 2022Byzantine Machine Learning Made Easy by Resilient Averaging of Momentumscited by 63
  18. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  19. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  20. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  21. 2021Adaptive Quantization of Model Updates for Communication-Efficient Federated Learningcited by 60
  22. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 60
  23. 2021Federated Learning with Fair Averagingcited by 59
  24. 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
  25. 2021A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variantscited by 57
  26. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  27. 2021Decentralized Federated Learning: Balancing Communication and Computing Costscited by 54
  28. 2023Performative Prediction: Past and Futurecited by 52
  29. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  30. 2021Consensus Control for Decentralized Deep Learningcited by 49
  31. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  32. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 48
  33. 2021Hierarchical Federated Learning with Quantization: Convergence Analysis and System Designcited by 47
  34. 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
  35. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  36. 2021Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivitycited by 43
  37. 2023Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspectivecited by 43
  38. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43

50 most cited of 1,140 All 1,140 (JSON)