mlsanityData to 14 Oct 2026

Dynamic Estimation of Learning Rates Using a Non-Linear Autoregressive Model

arXiv 2410.09943 ↗First posted 13 Oct 2024cs.LG math.OC stat.APcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce a new class of adaptive non-linear autoregressive (Nlar) models incorporating the concept of momentum, which dynamically estimate both the learning rates and momentum as the number of iterations increases. In our method, the growth of the gradients is controlled using a scaling (clipping) function, leading to stable convergence. Within this framework, we propose three distinct estimators for learning rates and provide theoretical proof of their convergence. We further demonstrate how these estimators underpin the development of effective Nlar optimizers. The performance of the proposed estimators and optimizers is rigorously evaluated through extensive experiments across several datasets and a reinforcement learning environment. The results highlight two key features of the Nlar optimizers: robust convergence despite variations in underlying parameters, including large initial learning rates, and strong adaptability with rapid convergence during the initial epochs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 31 references that are in the map. First posted on arXiv 13 Oct 2024.

Builds on

10 of 31 references.

References · 10

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  4. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 145
  5. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
  6. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
  7. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  8. 2020Adam^+: A Stochastic Method with Adaptive Variance Reductioncited by 18
  9. 2017Reinforcement Learning for Learning Rate Controlcited by 16
  10. 2020EAdam Optimizer: How ε Impact Adamcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning Rate-Free Reinforcement Learning: A Case for Model Selection with Non-Stationary Objectives
  2. 2024Dynamic Learning Rate for Deep Reinforcement Learning: A Bandit Approach
  3. 2025Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards
  4. 2026Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimator
  5. 2025Policy Gradient with Second Order Momentum
  6. 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation
  7. 2026Rethinking Entropy Minimization in Test-Time Adaptation for Autoregressive Models
  8. 2026Scalable Reinforcement Learning via Adaptive Batch Scaling