mlsanityData to 14 Oct 2026

Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization

arXiv 2411.15795 ↗First posted 24 Nov 2024cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Adaptive gradient methods have been increasingly adopted by deep learning community due to their fast convergence and reduced sensitivity to hyper-parameters. However, these methods come with limitations, such as increased memory requirements for elements like moving averages and a poorly understood convergence theory. To overcome these challenges, we introduce F-CMA, a Fast-Controlled Mini-batch Algorithm with a random reshuffling method featuring a sufficient decrease condition and a line-search procedure to ensure loss reduction per epoch, along with its deterministic proof of global convergence to a stationary point. To evaluate the F-CMA, we integrate it into conventional training protocols for classification tasks involving both convolutional neural networks and vision transformer models, allowing for a direct comparison with popular optimizers. Computational tests show significant improvements, including a decrease in the overall training time by up to 68%, an increase in per-epoch efficiency by up to 20%, and in model accuracy by up to 5%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 57 of its 95 references that are in the map. First posted on arXiv 24 Nov 2024.

Builds on

57 of 95 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  6. 2016Wide Residual Networkscited by 3,070
  7. 2017Decoupled Weight Decay Regularizationcited by 1,831
  8. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  9. 2016An overview of gradient descent optimization algorithmscited by 1,033
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  11. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  12. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
38 more
  1. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  2. 2021A Survey of Transformerscited by 401
  3. 2016Deep Learning without Poor Local Minimacited by 234
  4. 2019SGD: General Analysis and Improved Ratescited by 207
  5. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  6. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  7. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  8. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 142
  9. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  10. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  11. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  12. 2020Random Reshuffling: Simple Analysis with Vast Improvementscited by 115
  13. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  14. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  15. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 95
  16. 2016Variance Reduction for Faster Non-Convex Optimizationcited by 72
  17. 2015From error bounds to the complexity of first-order descent methods for convex functionscited by 43
  18. 2020Optimal and Practical Algorithms for Smooth and Strongly Convex Decentralized Optimizationcited by 43
  19. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  20. 2020Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networkscited by 40
  21. 2022On Almost Sure Convergence Rates of Stochastic Gradient Methodscited by 35
  22. 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 33
  23. 2021Proximal and Federated Random Reshufflingcited by 33
  24. 2015Probabilistic Line Searches for Stochastic Optimizationcited by 29
  25. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  26. 2015Convergence Rate of Incremental Gradient and Newton Methodscited by 27
  27. 2013Accelerated, Parallel and Proximal Coordinate Descentcited by 25
  28. 2013Distributed Coordinate Descent Method for Learning with Big Datacited by 23
  29. 2023Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methodscited by 23
  30. 2023Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reductioncited by 23
  31. 2015On the Convergence Rate of Incremental Aggregated Gradient Algorithmscited by 21
  32. 2018Accelerated Bregman Proximal Gradient Methods for Relatively Smooth Convex Optimizationcited by 20
  33. 2021Lower Bounds and Optimal Algorithms for Smooth and Strongly Convex Decentralized Optimization Over Time-Varying Networkscited by 19
  34. 2021Learning with Gradient Descent and Weakly Convex Lossescited by 16
  35. 2021Convergence of Random Reshuffling Under The Kurdyka-Lojasiewicz Inequalitycited by 16
  36. 2019Parabolic Approximation Line Search for DNNscited by 14
  37. 2022On the Convergence of AdaGrad(Norm) on Rd: Beyond Convexity, Non-Asymptotic Rate and Accelerationcited by 13
  38. 2016Surpassing Gradient Descent Provably: A Cyclic Incremental Method with Linear Convergence Ratecited by 12

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.