mlsanityData to 14 Oct 2026

To Raise or Not To Raise: The Autonomous Learning Rate Question

arXiv 2106.08767 ↗First posted 16 Jun 2021cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

There is a parameter ubiquitous throughout the deep learning world: learning rate. There is likewise a ubiquitous question: what should that learning rate be? The true answer to this question is often tedious and time consuming to obtain, and a great deal of arcane knowledge has accumulated in recent years over how to pick and modify learning rates to achieve optimal training performance. Moreover, the long hours spent carefully crafting the perfect learning rate can come to nothing the moment your network architecture, optimizer, dataset, or initial conditions change ever so slightly. But it need not be this way. We propose a new answer to the great learning rate question: the Autonomous Learning Rate Controller. Find it at https://github.com/fastestimator/ARC/tree/v2.0

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 57 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

31 of 57 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2016Densely Connected Convolutional Networkscited by 6,720
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
19 more
  1. 2016Wide Residual Networkscited by 3,070
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  3. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,630
  4. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  6. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  7. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  9. 2017Focal Loss for Dense Object Detectioncited by 752
  10. 2019CSPNet: A New Backbone that can Enhance Learning Capability of CNNcited by 381
  11. 2019Verified Uncertainty Calibrationcited by 309
  12. 2020Scaled-YOLOv4: Scaling Cross Stage Partial Networkcited by 211
  13. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 199
  14. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  15. 2017Neural Optimizer Search with Reinforcement Learningcited by 82
  16. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 69
  17. 2015Probabilistic Line Searches for Stochastic Optimizationcited by 29
  18. 2019Learning an Adaptive Learning Rate Schedulecited by 29
  19. 2019FastEstimator: A Deep Learning Library for Fast Prototyping and Productizationcited by 1

Built on by

Not cited yet.