# Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization

arXiv: https://arxiv.org/abs/2101.11075 · HTML: https://mlsanity.com/paper/2101.11075/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 26 Jan 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [signSGD: Compressed Optimisation for Non-Convex Problems](https://mlsanity.com/paper/1802.04434/) (2018)
- [fastMRI: An Open Dataset and Benchmarks for Accelerated MRI](https://mlsanity.com/paper/1811.08839/) (2018)
- [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017)
- [End-to-End Variational Networks for Accelerated MRI Reconstruction](https://mlsanity.com/paper/2004.06688/) (2020)
- [AdaGrad stepsizes: Sharp convergence over nonconvex landscapes](https://mlsanity.com/paper/1806.01811/) (2018)
- [A Simple Convergence Proof of Adam and Adagrad](https://mlsanity.com/paper/2003.02395/) (2020)
- [On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes](https://mlsanity.com/paper/1805.08114/) (2018)
- [Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients](https://mlsanity.com/paper/1705.07774/) (2017)
- [Sequence-to-Sequence Learning as Beam-Search Optimization](https://mlsanity.com/paper/1606.02960/) (2016)
- [A Sufficient Condition for Convergences of Adam and RMSProp](https://mlsanity.com/paper/1811.09358/) (2018)
- [Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers](https://mlsanity.com/paper/2007.01547/) (2020)
- [On Empirical Comparisons of Optimizers for Deep Learning](https://mlsanity.com/paper/1910.05446/) (2019)
- [On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization](https://mlsanity.com/paper/1808.05671/) (2018)
- [Online Adaptive Methods, Universality and Acceleration](https://mlsanity.com/paper/1809.02864/) (2018)
- [Scale-Free Algorithms for Online Linear Optimization](https://mlsanity.com/paper/1502.05744/) (2015)
- [The Power of Factorial Powers: New Parameter settings for (Stochastic) Optimization](https://mlsanity.com/paper/2006.01244/) (2020)
- [Almost sure convergence rates for Stochastic Gradient Descent and Stochastic Heavy Ball](https://mlsanity.com/paper/2006.07867/) (2020)
- [Dual Averaging is Surprisingly Effective for Deep Learning Optimization](https://mlsanity.com/paper/2010.10502/) (2020)
- [Offset Sampling Improves Deep Learning based Accelerated MRI Reconstructions by Exploiting Symmetry](https://mlsanity.com/paper/1912.01101/) (2019)

## Built on by (most cited first, 28 of 28)

- [Text2LIVE: Text-Driven Layered Image and Video Editing](https://mlsanity.com/paper/2204.02491/) (2022)
- [The Road Less Scheduled](https://mlsanity.com/paper/2405.15682/) (2024)
- [Prodigy: An Expeditiously Adaptive Parameter-Free Learner](https://mlsanity.com/paper/2306.06101/) (2023)
- [The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions](https://mlsanity.com/paper/2104.02710/) (2021)
- [Deep Symbolic Learning: Discovering Symbols and Rules from Perceptions](https://mlsanity.com/paper/2208.11561/) (2022)
- [Noise-aware Physics-informed Machine Learning for Robust PDE Discovery](https://mlsanity.com/paper/2206.12901/) (2022)
- [Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case](https://mlsanity.com/paper/2307.11782/) (2023)
- [Contrastive Pretraining for Echocardiography Segmentation with Limited Data](https://mlsanity.com/paper/2201.07219/) (2022)
- [Shortcuts and Identifiability in Concept-based Models from a Neuro-Symbolic Lens](https://mlsanity.com/paper/2502.11245/) (2025)
- [Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods](https://mlsanity.com/paper/2510.10777/) (2025)
- [Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Method](https://mlsanity.com/paper/2206.06900/) (2022)
- [AirTrack: Onboard Deep Learning Framework for Long-Range Aircraft Detection and Tracking](https://mlsanity.com/paper/2209.12849/) (2022)
- [SD-LayerNet: Semi-supervised retinal layer segmentation in OCT using disentangled representation with anatomical priors](https://mlsanity.com/paper/2207.00458/) (2022)
- [Deciphering Hate: Identifying Hateful Memes and Their Targets](https://mlsanity.com/paper/2403.10829/) (2024)
- [Wasserstein Adversarial Transformer for Cloud Workload Prediction](https://mlsanity.com/paper/2203.06501/) (2022)
- [Skeleton-based Human Action Recognition via Convolutional Neural Networks (CNN)](https://mlsanity.com/paper/2301.13360/) (2023)
- [Robust vertebra identification using simultaneous node and edge predicting Graph Neural Networks](https://mlsanity.com/paper/2308.02509/) (2023)
- [How Much Context Does My Attention-Based ASR System Need?](https://mlsanity.com/paper/2310.15672/) (2023)
- [SSM-Net: feature learning for Music Structure Analysis using a Self-Similarity-Matrix based loss](https://mlsanity.com/paper/2211.08141/) (2022)
- [Leveraging Cross-Utterance Context For ASR Decoding](https://mlsanity.com/paper/2306.16903/) (2023)
- [A theoretical and empirical study of new adaptive algorithms with additional momentum steps and shifted updates for stochastic non-convex optimization](https://mlsanity.com/paper/2110.08531/) (2021)
- [Codec at SemEval-2022 Task 5: Multi-Modal Multi-Transformer Misogynous Meme Classification Framework](https://mlsanity.com/paper/2206.07190/) (2022)
- [Promoting Exploration in Memory-Augmented Adam using Critical Momenta](https://mlsanity.com/paper/2307.09638/) (2023)
- [Regularized Adaptive Momentum Dual Averaging with an Efficient Inexact Subproblem Solver for Training Structured Neural Network](https://mlsanity.com/paper/2403.14398/) (2024)
- [Prompt-driven Universal Model for View-Agnostic Echocardiography Analysis](https://mlsanity.com/paper/2404.05916/) (2024)
- [Self-Train Before You Transcribe](https://mlsanity.com/paper/2406.12937/) (2024)
- [Retrieval Augmented Enhanced Dual Co-Attention Framework for Target Aware Multimodal Bengali Hateful Meme Detection](https://mlsanity.com/paper/2602.19212/) (2026)
- [Minimax Optimality of Shampoo for Nonstationary Stochastic Optimization](https://mlsanity.com/paper/2610.16923/) (2026)

## Abstract (arXiv)

We introduce MADGRAD, a novel optimization method in the family of AdaGrad adaptive gradient methods. MADGRAD shows excellent performance on deep learning optimization problems from multiple fields, including classification and image-to-image tasks in vision, and recurrent and bidirectionally-masked models in natural language processing. For each of these tasks, MADGRAD matches or outperforms both SGD and ADAM in test set performance, even on problems for which adaptive methods normally perform poorly.