mlsanityData to 14 Oct 2026

Accelerated Gradient Descent for Faster Convergence with Minimal Overhead

arXiv 2605.16017 ↗First posted 15 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we present CT-AGD (Curvature-Tuned Accelerated Gradient Descent), an optimization method for non-convex optimization problems in deep learning training tasks. CT-AGD is a general boosting procedure that accelerates first-order methods by explicitly capturing the local curvature using finite-difference quotients, and the development of heuristics aimed at mitigating noise and bias introduced by stochastic mini-batch training. CT-AGD has a comparable storage and computational overhead as adaptive gradient methods such as Adam. Our extensive experiments demonstrate that CT-AGD achieves the same level of accuracy as the baseline first-order methods, yet reduces the required training epochs by 33% on average.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

27 of 47 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2016Wide Residual Networkscited by 3,070
  4. 2021Convex optimizationcited by 2,515
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  6. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  7. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  8. 2016An overview of gradient descent optimization algorithmscited by 1,033
  9. 2019On the Convergence of Adam and Beyondcited by 841
  10. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  11. 2019Green AIcited by 627
  12. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
15 more
  1. 2014Convex Optimization: Algorithms and Complexitycited by 575
  2. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  3. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  4. 2017How to Escape Saddle Points Efficientlycited by 285
  5. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  6. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  7. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  8. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
  9. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  10. 2014A Stochastic Quasi-Newton Method for Large-Scale Optimizationcited by 69
  11. 2020Practical Quasi-Newton Methods for Training Deep Neural Networkscited by 68
  12. 2018A Progressive Batching L-BFGS Method for Machine Learningcited by 45
  13. 2016Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimizationcited by 37
  14. 2016The Power of Normalization: Faster Evasion of Saddle Pointscited by 29
  15. 2017A Generic Approach for Escaping Saddle pointscited by 13

Built on by

Not cited yet.