Data to 7 Oct 2026

Cyclical Learning Rates for Training Neural Networks

arXiv 1506.01186 ↗First posted 3 Jun 2015cs.CV cs.LG cs.NEcited by 627

Not on the map. Why?

Abstract (arXiv)

It is known that the learning rate is the most important hyper-parameter to tune for training deep neural networks. This paper describes a new method for setting the learning rate, named cyclical learning rates, which practically eliminates the need to experimentally find the best values and schedule for the global learning rates. Instead of monotonically decreasing the learning rate, this method lets the learning rate cyclically vary between reasonable boundary values. Training with cyclical learning rates instead of fixed values achieves improved classification accuracy without a need to tune and often in fewer iterations. This paper also describes a simple way to estimate "reasonable bounds" -- linearly increasing the learning rate of the network for a few epochs. In addition, cyclical learning rates are demonstrated on the CIFAR-10 and CIFAR-100 datasets with ResNets, Stochastic Depth networks, and DenseNets, and the ImageNet dataset with the AlexNet and GoogLeNet architectures. These are practical tools for everyone who trains neural networks.

Key facts. Cited by 627 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 139; 2022: 132; 2023: 124; 2024: 97; 2025: 85; 2026: 49.

627 papers.

Sort:

By area:

2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
2022Matryoshka Representation Learningcited by 268
2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
2023SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detectioncited by 83
2022MapQA: A Dataset for Question Answering on Choropleth Mapscited by 82
2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
2021Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resourcescited by 71
2021Learning from Counterfactual Links for Link Predictioncited by 69
2023Cross Modal Transformer: Towards Fast and Robust 3D Object Detectioncited by 68
2023Can Neural Network Memorization Be Localized?cited by 66
2021Object DGCNN: 3D Object Detection using Dynamic Graphscited by 58

38 more

2024IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detectioncited by 57
2022HYDRA: Competing convolutional kernels for fast and accurate time series classificationcited by 53
2023FFCV: Accelerating Training by Removing Data Bottleneckscited by 49
2021SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environmentscited by 48
2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
2021A Survey on Ensemble Learning under the Era of Deep Learningcited by 47
2021Random Noise Defense Against Query-Based Black-Box Attackscited by 46
2023Robust Principles: Architectural Design Principles for Adversarially Robust CNNscited by 46
2022A Survey of Robust Adversarial Training in Pattern Recognition: Fundamental, Theory, and Methodologiescited by 44
2021Data-driven discovery of Green's functions with human-understandable deep learningcited by 43
2021Subspace Adversarial Trainingcited by 41
2024Instance-Adaptive and Geometric-Aware Keypoint Learning for Category-Level 6D Object Pose Estimationcited by 41
2021Boosting Fast Adversarial Training with Learnable Adversarial Initializationcited by 40
2022Towards Long-Tailed 3D Detectioncited by 40
2021Deep Learning-based Patient Re-identification Is able to Exploit the Biometric Nature of Medical Chest X-ray Datacited by 38
2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
2022Contrastive Meta Learning with Behavior Multiplicity for Recommendationcited by 37
2022Automated Dynamic Algorithm Configurationcited by 37
20213KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Augmentationscited by 36
2021Fine-tuning wav2vec2 for speaker recognitioncited by 35
2021Sound-Guided Semantic Image Manipulationcited by 35
2023Deep networks for system identification: a Surveycited by 35
2022CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learningcited by 34
2022Prior-Guided Adversarial Initialization for Fast Adversarial Trainingcited by 34
2023Improving Fast Adversarial Training with Prior-Guided Knowledgecited by 34
2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
2025MAT-Agent: Adaptive Multi-Agent Training Optimizationcited by 34
2021End-to-end Prostate Cancer Detection in bpMRI via 3D CNNs: Effects of Attention Mechanisms, Clinical Priori and Decoupled False Positive Reductioncited by 32
2022Constrained Monotonic Neural Networkscited by 31
2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
2021Extremely Lightweight Quantization Robust Real-Time Single-Image Super Resolution for Mobile Devicescited by 30
2024Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budgetcited by 30
2025Accelerating Neural Network Training: An Analysis of the AlgoPerf Competitioncited by 30
2021You Only Learn Once: Universal Anatomical Landmark Detectioncited by 29
2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
2024Scaling Law with Learning Rate Annealingcited by 29
2023Detecting As Labeling: Rethinking LiDAR-camera Fusion in 3D Object Detectioncited by 28
2021OODformer: Out-Of-Distribution Detection Transformercited by 27

50 most cited of 627 All 627 (JSON)