# Tune smarter not harder: A principled approach to tuning learning rates for shallow nets

arXiv: https://arxiv.org/abs/2003.09844 · HTML: https://mlsanity.com/paper/2003.09844/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2020.

## Built on by (most cited first, 2 of 2)

- [Randomness Helps Rigor: A Probabilistic Learning Rate Scheduler Bridging Theory and Deep Learning Practice](https://mlsanity.com/paper/2407.07613/) (2024)
- [Learning Rate Optimization for Deep Neural Networks Using Lipschitz Bandits](https://mlsanity.com/paper/2409.09783/) (2024)

## Abstract (arXiv)

Effective hyper-parameter tuning is essential to guarantee the performance that neural networks have come to be known for. In this work, a principled approach to choosing the learning rate is proposed for shallow feedforward neural networks. We associate the learning rate with the gradient Lipschitz constant of the objective to be minimized while training. An upper bound on the mentioned constant is derived and a search algorithm, which always results in non-divergent traces, is proposed to exploit the derived bound. It is shown through simulations that the proposed search method significantly outperforms the existing tuning methods such as Tree Parzen Estimators (TPE). The proposed method is applied to three different existing applications: a) channel estimation in OFDM systems, b) prediction of the exchange currency rates and c) offset estimation in OFDM receivers, and it is shown to pick better learning rates than the existing methods using the same or lesser compute power.