# Ranger21: a synergistic deep learning optimizer

arXiv: https://arxiv.org/abs/2106.13731 · HTML: https://mlsanity.com/paper/2106.13731/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 25 Jun 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers](https://mlsanity.com/paper/2007.01547/) (2020)
- [Gradient Centralization: A New Optimization Technique for Deep Neural Networks](https://mlsanity.com/paper/2004.01461/) (2020)
- [On the adequacy of untuned warmup for adaptive optimization](https://mlsanity.com/paper/1910.04209/) (2019)
- [Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization](https://mlsanity.com/paper/2103.17182/) (2021)
- [Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule](https://mlsanity.com/paper/2003.03977/) (2020)
- [Adaptive Learning Rate Clipping Stabilizes Learning](https://mlsanity.com/paper/1906.09060/) (2019)
- [Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks](https://mlsanity.com/paper/2103.06583/) (2021)

## Built on by (most cited first, 22 of 22)

- [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023)
- [blob loss: instance imbalance aware loss functions for semantic segmentation](https://mlsanity.com/paper/2205.08209/) (2022)
- [MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysis](https://mlsanity.com/paper/2408.07773/) (2024)
- [Spatial-Temporal Interactive Dynamic Graph Convolution Network for Traffic Forecasting](https://mlsanity.com/paper/2205.08689/) (2022)
- [Where is VALDO? VAscular Lesions Detection and segmentatiOn challenge at MICCAI 2021](https://mlsanity.com/paper/2208.07167/) (2022)
- [Variational Probabilistic Fusion Network for RGB-T Semantic Segmentation](https://mlsanity.com/paper/2307.08536/) (2023)
- [Deepbet: Fast brain extraction of T1-weighted MRI using Convolutional Neural Networks](https://mlsanity.com/paper/2308.07003/) (2023)
- [Deep Quality Estimation: Creating Surrogate Models for Human Quality Ratings](https://mlsanity.com/paper/2205.10355/) (2022)
- [Knowledge Sharing via Domain Adaptation in Customs Fraud Detection](https://mlsanity.com/paper/2201.06759/) (2022)
- [Pre-Training LLMs on a budget: A comparison of three optimizers](https://mlsanity.com/paper/2507.08472/) (2025)
- [Temporal Fusion Transformers for Streamflow Prediction: Value of Combining Attention with Recurrence](https://mlsanity.com/paper/2305.12335/) (2023)
- [Investigating Neural Architectures by Synthetic Dataset Design](https://mlsanity.com/paper/2204.11045/) (2022)
- [Insights on Galaxy Evolution from Interpretable Sparse Feature Networks](https://mlsanity.com/paper/2501.00089/) (2024)
- [Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules](https://mlsanity.com/paper/2509.18396/) (2025)
- [Momentum Capsule Networks](https://mlsanity.com/paper/2201.11091/) (2022)
- [Automatic Bounding Box Annotation with Small Training Data Sets for Industrial Manufacturing](https://mlsanity.com/paper/2206.00280/) (2022)
- [Fusion Matrix Prompt Enhanced Self-Attention Spatial-Temporal Interactive Traffic Forecasting Framework](https://mlsanity.com/paper/2410.09356/) (2024)
- [Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Tricks](https://mlsanity.com/paper/2503.02147/) (2025)
- [Pose and Facial Expression Transfer by using StyleGAN](https://mlsanity.com/paper/2504.13021/) (2025)
- [Dataset-Aware Cold-Start Active Learning for Annotation-Efficient 3D Medical Image Segmentation](https://mlsanity.com/paper/2606.20765/) (2026)
- [OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers](https://mlsanity.com/paper/2607.04033/) (2026)
- [Enhancing Table Structure Recognition via Bounding Box Guidance](https://mlsanity.com/paper/2609.08705/) (2026)

## Abstract (arXiv)

As optimizers are critical to the performances of neural networks, every year a large number of papers innovating on the subject are published. However, while most of these publications provide incremental improvements to existing algorithms, they tend to be presented as new optimizers rather than composable algorithms. Thus, many worthwhile improvements are rarely seen out of their initial publication. Taking advantage of this untapped potential, we introduce Ranger21, a new optimizer which combines AdamW with eight components, carefully selected after reviewing and testing ideas from the literature. We found that the resulting optimizer provides significantly improved validation accuracy and training speed, smoother training curves, and is even able to train a ResNet50 on ImageNet2012 without Batch Normalization layers. A problem on which AdamW stays systematically stuck in a bad initial state.