# Layer-Specific Adaptive Learning Rates for Deep Networks

arXiv: https://arxiv.org/abs/1510.04609 · HTML: https://mlsanity.com/paper/1510.04609/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2015.

## Built on by (most cited first, 19 of 19)

- [Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/2112.07869/) (2021)
- [Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training](https://mlsanity.com/paper/2312.00359/) (2023)
- [Leveraging the two timescale regime to demonstrate convergence of neural networks](https://mlsanity.com/paper/2304.09576/) (2023)
- [LAFS: Landmark-based Facial Self-supervised Learning for Face Recognition](https://mlsanity.com/paper/2403.08161/) (2024)
- [Feature learning as alignment: a structural property of gradient descent in non-linear neural networks](https://mlsanity.com/paper/2402.05271/) (2024)
- [Learning Rate Curriculum](https://mlsanity.com/paper/2205.09180/) (2022)
- [Continuous Visual Autoregressive Generation via Score Maximization](https://mlsanity.com/paper/2505.07812/) (2025)
- [SHREC 2022: pothole and crack detection in the road pavement using images and RGB-D data](https://mlsanity.com/paper/2205.13326/) (2022)
- [MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters](https://mlsanity.com/paper/2402.02342/) (2024)
- [Closed-Form Last Layer Optimization](https://mlsanity.com/paper/2510.04606/) (2025)
- [Optimizing Personalized Federated Learning through Adaptive Layer-Wise Learning](https://mlsanity.com/paper/2412.07062/) (2024)
- [Unsupervised Model Drift Estimation with Batch Normalization Statistics for Dataset Shift Detection and Model Selection](https://mlsanity.com/paper/2107.00191/) (2021)
- [Faster Convergence for Transformer Fine-tuning with Line Search Methods](https://mlsanity.com/paper/2403.18506/) (2024)
- [A Hessian-informed hyperparameter optimization for differential learning rate](https://mlsanity.com/paper/2501.06954/) (2025)
- [Rethinking Neural Network Learning Rates: A Stackelberg Perspective](https://mlsanity.com/paper/2605.15530/) (2026)
- [Input Normalized Stochastic Gradient Descent Training of Deep Neural Networks](https://mlsanity.com/paper/2212.09921/) (2022)
- [Improving Reliability of Fine-tuning with Block-wise Optimisation](https://mlsanity.com/paper/2301.06133/) (2023)
- [Transformer-based interpretable multi-modal data fusion for skin lesion classification](https://mlsanity.com/paper/2304.14505/) (2023)
- [A Closed-Form Upper Bound for Admissible Learning-Rate Steps in Belief-Space Dynamics](https://mlsanity.com/paper/2605.06741/) (2026)

## Abstract (arXiv)

The increasing complexity of deep learning architectures is resulting in training time requiring weeks or even months. This slow training is due in part to vanishing gradients, in which the gradients used by back-propagation are extremely large for weights connecting deep layers (layers near the output layer), and extremely small for shallow layers (near the input layer); this results in slow learning in the shallow layers. Additionally, it has also been shown that in highly non-convex problems, such as deep neural networks, there is a proliferation of high-error low curvature saddle points, which slows down learning dramatically. In this paper, we attempt to overcome the two above problems by proposing an optimization method for training deep neural networks which uses learning rates which are both specific to each layer in the network and adaptive to the curvature of the function, increasing the learning rate at low curvature points. This enables us to speed up learning in the shallow layers of the network and quickly escape high-error low curvature saddle points. We test our method on standard image classification datasets such as MNIST, CIFAR10 and ImageNet, and demonstrate that our method increases accuracy as well as reduces the required training time over standard algorithms.