mlsanityData to 14 Oct 2026

Ranger21: a synergistic deep learning optimizer

arXiv 2106.13731 ↗First posted 25 Jun 2021cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

As optimizers are critical to the performances of neural networks, every year a large number of papers innovating on the subject are published. However, while most of these publications provide incremental improvements to existing algorithms, they tend to be presented as new optimizers rather than composable algorithms. Thus, many worthwhile improvements are rarely seen out of their initial publication. Taking advantage of this untapped potential, we introduce Ranger21, a new optimizer which combines AdamW with eight components, carefully selected after reviewing and testing ideas from the literature. We found that the resulting optimizer provides significantly improved validation accuracy and training speed, smoother training curves, and is even able to train a ResNet50 on ImageNet2012 without Batch Normalization layers. A problem on which AdamW stays systematically stuck in a bad initial state.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 16 of its 27 references that are in the map. First posted on arXiv 25 Jun 2021.

Builds on

16 of 27 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  7. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  8. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  9. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  10. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 119
  11. 2020Gradient Centralization: A New Optimization Technique for Deep Neural Networkscited by 82
  12. 2019On the adequacy of untuned warmup for adaptive optimizationcited by 35
4 more
  1. 2021Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalizationcited by 28
  2. 2020Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedulecited by 25
  3. 2019Adaptive Learning Rate Clipping Stabilizes Learningcited by 6
  4. 2021Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 4; 2024: 3; 2025: 4; 2026: 3.

22 papers.

Sort:

  1. 2023SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challengecited by 56
  2. 2022blob loss: instance imbalance aware loss functions for semantic segmentationcited by 18
  3. 2024MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysiscited by 15
  4. 2022Spatial-Temporal Interactive Dynamic Graph Convolution Network for Traffic Forecastingcited by 14
  5. 2022Where is VALDO? VAscular Lesions Detection and segmentatiOn challenge at MICCAI 2021cited by 14
  6. 2023Variational Probabilistic Fusion Network for RGB-T Semantic Segmentationcited by 7
  7. 2023Deepbet: Fast brain extraction of T1-weighted MRI using Convolutional Neural Networkscited by 7
  8. 2022Deep Quality Estimation: Creating Surrogate Models for Human Quality Ratingscited by 6
  9. 2022Knowledge Sharing via Domain Adaptation in Customs Fraud Detectioncited by 5
  10. 2025Pre-Training LLMs on a budget: A comparison of three optimizerscited by 5
  11. 2023Temporal Fusion Transformers for Streamflow Prediction: Value of Combining Attention with Recurrencecited by 4
  12. 2022Investigating Neural Architectures by Synthetic Dataset Designcited by 3
10 more
  1. 2024Insights on Galaxy Evolution from Interpretable Sparse Feature Networkscited by 2
  2. 2025Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rulescited by 2
  3. 2022Momentum Capsule Networkscited by 0
  4. 2022Automatic Bounding Box Annotation with Small Training Data Sets for Industrial Manufacturingcited by 0
  5. 2024Fusion Matrix Prompt Enhanced Self-Attention Spatial-Temporal Interactive Traffic Forecasting Frameworkcited by 0
  6. 2025Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Trickscited by 0
  7. 2025Pose and Facial Expression Transfer by using StyleGANcited by 0
  8. 2026Dataset-Aware Cold-Start Active Learning for Annotation-Efficient 3D Medical Image Segmentationcited by 0
  9. 2026OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizerscited by 0
  10. 2026Enhancing Table Structure Recognition via Bounding Box Guidancecited by 0