mlsanityData to 7 Oct 2026

Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates

arXiv 2206.00832 ↗First posted 2 Jun 2022cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Benchmarking the tradeoff between neural network accuracy and training time is computationally expensive. Here we show how a multiplicative cyclic learning rate schedule can be used to construct a tradeoff curve in a single training run. We generate cyclic tradeoff curves for combinations of training methods such as Blurpool, Channels Last, Label Smoothing and MixUp, and highlight how these cyclic tradeoff curves can be used to evaluate the effects of algorithmic choices on network training efficiency.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

18 of 24 references.

References · 18

  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2019When Does Label Smoothing Help?cited by 985
  6. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  7. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  8. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  9. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  10. 2020fastai: A Layered API for Deep Learningcited by 183
  11. 2020Learning the Pareto Front with Hypernetworkscited by 122
  12. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
6 more
  1. 2020The Early Phase of Neural Network Trainingcited by 114
  2. 2019Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergencecited by 54
  3. 2020Controllable Pareto Multi-Task Learningcited by 51
  4. 2021Super-Acceleration with Cyclical Step-sizescited by 14
  5. 2019Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networkscited by 12
  6. 2022General Cyclical Training of Neural Networkscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 1; 2025: 1.

5 papers.

  1. 2024The Road Less Scheduledcited by 158
  2. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  3. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  4. 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 33
  5. 2025When, Where and Why to Average Weights?cited by 6