mlsanityData to 7 Oct 2026

Probabilistic Line Searches for Stochastic Optimization

arXiv 1502.02846 ↗First posted 10 Feb 2015cs.LG math.OC stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

In deterministic optimization, line searches are a standard tool ensuring stability and efficiency. Where only stochastic gradients are available, no direct equivalent has so far been formulated, because uncertain gradients do not allow for a strict sequence of decisions collapsing the search space. We construct a probabilistic line search by combining the structure of existing deterministic methods with notions from Bayesian optimization. Our method retains a Gaussian process surrogate of the univariate optimization objective, and uses a probabilistic belief over the Wolfe conditions to monitor the descent. The algorithm has very low computational cost, and no user-controlled parameters. Experiments show that it effectively removes the need to define a learning rate for stochastic gradient descent.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Feb 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 1; 2023: 3; 2024: 5; 2025: 3; 2026: 3.

29 papers.

Sort:

  1. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  2. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 33
  3. 2022Local Bayesian optimization via maximizing probability of descentcited by 29
  4. 2023Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Modelscited by 15
  5. 2021AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Flycited by 14
  6. 2021Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networkscited by 7
  7. 2021Stochastic Gradient Line Bayesian Optimization for Efficient Noise-Robust Optimization of Parameterized Quantum Circuitscited by 7
  8. 2021KOALA: A Kalman Optimization Algorithm with Loss Adaptivitycited by 6
  9. 2026BayeSQP: Bayesian Optimization through Sequential Quadratic Programmingcited by 5
  10. 2021Empirically explaining SGD from a line search perspectivecited by 2
  11. 2024Faster Convergence for Transformer Fine-tuning with Line Search Methodscited by 1
  12. 2024Improving Line Search Methods for Large Scale Neural Network Trainingcited by 1
17 more
  1. 2024No learning rates needed: Introducing SALSA -- Stable Armijo Line Search Adaptationcited by 1
  2. 2025Bayesian autoregression to optimize temporal Mat'ern kernel Gaussian process hyperparameterscited by 1
  3. 2021A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimizationcited by 0
  4. 2021Discriminative Bayesian filtering lends momentum to the stochastic Newton method for minimizing log-convex functionscited by 0
  5. 2021GOALS: Gradient-Only Approximations for Line Searches Towards Robust and Consistent Training of Deep Neural Networkscited by 0
  6. 2021To Raise or Not To Raise: The Autonomous Learning Rate Questioncited by 0
  7. 2021Black Box Probabilistic Numericscited by 0
  8. 2021Using a one dimensional parabolic model of the full-batch loss to estimate learning rates during trainingcited by 0
  9. 2021Bolstering Stochastic Gradient Descent with Model Buildingcited by 0
  10. 2023Layer-wise Adaptive Step-Sizes for Stochastic First-Order Methods for Deep Learningcited by 0
  11. 2023Learning-Rate-Free Learning: Dissecting D-Adaptation and Probabilistic Line Searchcited by 0
  12. 2024Painless Federated Learning: An Interplay of Line-Search and Extrapolationcited by 0
  13. 2024Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimizationcited by 0
  14. 2025Stochastic Subspace Descent Accelerated via Bi-fidelity Line Searchcited by 0
  15. 2025Information Preserving Line Search via Bayesian Optimizationcited by 0
  16. 2026The cost of useful natural gradient updatescited by 0
  17. 2026Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuningcited by 0