# QLABGrad: a Hyperparameter-Free and Convergence-Guaranteed Scheme for Deep Learning

arXiv: https://arxiv.org/abs/2302.00252 · HTML: https://mlsanity.com/paper/2302.00252/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 27 references that are in the map.
- First posted on arXiv 1 Feb 2023.

## Builds on (most cited first, 8 of 8)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Online Learning Rate Adaptation with Hypergradient Descent](https://mlsanity.com/paper/1703.04782/) (2017)
- [Escaping Saddle Points with Adaptive Gradient Methods](https://mlsanity.com/paper/1901.09149/) (2019)
- [L4: Practical loss-based stepsize adaptation for deep learning](https://mlsanity.com/paper/1802.05074/) (2018)

## Built on by (most cited first, 6 of 6)

- [The Ninth NTIRE 2024 Efficient Super-Resolution Challenge Report](https://mlsanity.com/paper/2404.10343/) (2024)
- [NTIRE 2024 Challenge on Low Light Image Enhancement: Methods and Results](https://mlsanity.com/paper/2404.14248/) (2024)
- [Gradient descent with generalized Newton's method](https://mlsanity.com/paper/2407.02772/) (2024)
- [Multi-modal Cross-domain Self-supervised Pre-training for fMRI and EEG Fusion](https://mlsanity.com/paper/2409.19130/) (2024)
- [Adaptive parameter-efficient fine-tuning via Hessian-informed subset selection](https://mlsanity.com/paper/2505.12579/) (2025)
- [ZENITH: Automated Gradient Norm Informed Stochastic Optimization](https://mlsanity.com/paper/2601.15212/) (2026)

## Abstract (arXiv)

The learning rate is a critical hyperparameter for deep learning tasks since it determines the extent to which the model parameters are updated during the learning course. However, the choice of learning rates typically depends on empirical judgment, which may not result in satisfactory outcomes without intensive try-and-error experiments. In this study, we propose a novel learning rate adaptation scheme called QLABGrad. Without any user-specified hyperparameter, QLABGrad automatically determines the learning rate by optimizing the Quadratic Loss Approximation-Based (QLAB) function for a given gradient descent direction, where only one extra forward propagation is required. We theoretically prove the convergence of QLABGrad with a smooth Lipschitz condition on the loss function. Experiment results on multiple architectures, including MLP, CNN, and ResNet, on MNIST, CIFAR10, and ImageNet datasets, demonstrate that QLABGrad outperforms various competing schemes for deep learning.