# Parabolic Approximation Line Search for DNNs

arXiv: https://arxiv.org/abs/1903.11991 · HTML: https://mlsanity.com/paper/1903.11991/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2019.

## Built on by (most cited first, 14 of 14)

- [Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models](https://mlsanity.com/paper/2306.12747/) (2023)
- [Stepping on the Edge: Curvature Aware Learning Rate Tuners](https://mlsanity.com/paper/2407.06183/) (2024)
- [KOALA: A Kalman Optimization Algorithm with Loss Adaptivity](https://mlsanity.com/paper/2107.03331/) (2021)
- [On the Interplay Between Stepsize Tuning and Progressive Sharpening](https://mlsanity.com/paper/2312.00209/) (2023)
- [Empirically explaining SGD from a line search perspective](https://mlsanity.com/paper/2103.17132/) (2021)
- [Convergence Conditions for Stochastic Line Search Based Optimization of Over-parametrized Models](https://mlsanity.com/paper/2408.03199/) (2024)
- [Effectively Leveraging Momentum Terms in Stochastic Line Search Frameworks for Fast Optimization of Finite-Sum Problems](https://mlsanity.com/paper/2411.07102/) (2024)
- [Adaptive Learning Rate and Momentum for Training Deep Neural Networks](https://mlsanity.com/paper/2106.11548/) (2021)
- [Using a one dimensional parabolic model of the full-batch loss to estimate learning rates during training](https://mlsanity.com/paper/2108.13880/) (2021)
- [Bolstering Stochastic Gradient Descent with Model Building](https://mlsanity.com/paper/2111.07058/) (2021)
- [An extended asymmetric sigmoid with Perceptron (SIGTRON) for imbalanced linear classification](https://mlsanity.com/paper/2312.16043/) (2023)
- [Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization](https://mlsanity.com/paper/2411.15795/) (2024)
- [ZENITH: Automated Gradient Norm Informed Stochastic Optimization](https://mlsanity.com/paper/2601.15212/) (2026)
- [Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam](https://mlsanity.com/paper/2607.03998/) (2026)

## Abstract (arXiv)

A major challenge in current optimization research for deep learning is to automatically find optimal step sizes for each update step. The optimal step size is closely related to the shape of the loss in the update step direction. However, this shape has not yet been examined in detail. This work shows empirically that the batch loss over lines in negative gradient direction is mostly convex locally and well suited for one-dimensional parabolic approximations. By exploiting this parabolic property we introduce a simple and robust line search approach, which performs loss-shape dependent update steps. Our approach combines well-known methods such as parabolic approximation, line search and conjugate gradient, to perform efficiently. It surpasses other step size estimating methods and competes with common optimization methods on a large variety of experiments without the need of hand-designed step size schedules. Thus, it is of interest for objectives where step-size schedules are unknown or do not perform well. Our extensive evaluation includes multiple comprehensive hyperparameter grid searches on several datasets and architectures. Finally, we provide a general investigation of exact line searches in the context of batch losses and exact losses, including their relation to our line search approach.