# A Theoretical and Experimental Study of a Novel Adaptive Learning Algorithm

arXiv: https://arxiv.org/abs/2605.29273 · HTML: https://mlsanity.com/paper/2605.29273/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 9 of 9)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems](https://mlsanity.com/paper/2111.02801/) (2021)
- [Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://mlsanity.com/paper/1902.09843/) (2019)
- [New Convergence Aspects of Stochastic Gradient Algorithms](https://mlsanity.com/paper/1811.12403/) (2018)
- [Aiming towards the minimizers: fast convergence of SGD for overparametrized problems](https://mlsanity.com/paper/2306.02601/) (2023)
- [Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate](https://mlsanity.com/paper/1805.07557/) (2018)
- [Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling](https://mlsanity.com/paper/2506.12543/) (2025)
- [When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalities](https://mlsanity.com/paper/2604.10787/) (2026)

## Abstract (arXiv)

A crucial component of machine learning algorithms is minimizing loss functions with less computational cost and less oscillations. While adaptive learning rate-based optimizers have been widely used for real-world tasks, they do not guarantee convergence, which is why AMSGrad was later introduced to investigate the non-convergence behaviour of Adam. In this paper, popular adaptive optimization methods like Adam and AMSGrad are critically reviewed with an emphasis on their fundamental design concepts. To address limitations of the above mentioned optimizers, a new optimizer variant, C-Adam, is proposed based on the line of sight approach. A theoretical proof for convergence is also provided and the optimizer is validated through a number of real-life based numerical experiments.