# A Simple Dynamic Learning Rate Tuning Algorithm For Automated Training of DNNs

arXiv: https://arxiv.org/abs/1910.11605 · HTML: https://mlsanity.com/paper/1910.11605/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2019.

## Built on by (most cited first, 7 of 7)

- [Few Shot Activity Recognition Using Variational Inference](https://mlsanity.com/paper/2108.08990/) (2021)
- [LRWR: Large-Scale Benchmark for Lip Reading in Russian language](https://mlsanity.com/paper/2109.06692/) (2021)
- [Towards Efficient and Data Agnostic Image Classification Training Pipeline for Embedded Systems](https://mlsanity.com/paper/2108.07049/) (2021)
- [Semantic Depth Matters: Explaining Errors of Deep Vision Networks through Perceived Class Similarities](https://mlsanity.com/paper/2504.09956/) (2025)
- [CXR-CML: Improved zero-shot classification of long-tailed multi-label diseases in Chest X-Rays](https://mlsanity.com/paper/2507.19398/) (2025)
- [Reducing Overtreatment of Indeterminate Thyroid Nodules Using a Multimodal Deep Learning Model](https://mlsanity.com/paper/2409.19171/) (2024)
- [Transfer Learning for Keypoint Detection in Low-Resolution Thermal TUG Test Images](https://mlsanity.com/paper/2501.18453/) (2025)

## Abstract (arXiv)

Training neural networks on image datasets generally require extensive experimentation to find the optimal learning rate regime. Especially, for the cases of adversarial training or for training a newly synthesized model, one would not know the best learning rate regime beforehand. We propose an automated algorithm for determining the learning rate trajectory, that works across datasets and models for both natural and adversarial training, without requiring any dataset/model specific tuning. It is a stand-alone, parameterless, adaptive approach with no computational overhead. We theoretically discuss the algorithm's convergence behavior. We empirically validate our algorithm extensively. Our results show that our proposed approach consistently achieves top-level accuracy compared to SOTA baselines in the literature in natural as well as adversarial training.