mlsanityData to 7 Oct 2026

Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization

arXiv 2101.11075 ↗First posted 26 Jan 2021cs.LG cs.AI math.OCcited by 27

Not on the map. Why?

Abstract (arXiv)

We introduce MADGRAD, a novel optimization method in the family of AdaGrad adaptive gradient methods. MADGRAD shows excellent performance on deep learning optimization problems from multiple fields, including classification and image-to-image tasks in vision, and recurrent and bidirectionally-masked models in natural language processing. For each of these tasks, MADGRAD matches or outperforms both SGD and ADAM in test set performance, even on problems for which adaptive methods normally perform poorly.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 26 Jan 2021.

Builds on

26 of 38 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  7. 2019On the Convergence of Adam and Beyondcited by 834
  8. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  9. 2018fastMRI: An Open Dataset and Benchmarks for Accelerated MRIcited by 464
  10. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  11. 2020End-to-End Variational Networks for Accelerated MRI Reconstructioncited by 201
  12. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
14 more
  1. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  2. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  3. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
  4. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  5. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  6. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
  7. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 99
  8. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  9. 2018Online Adaptive Methods, Universality and Accelerationcited by 51
  10. 2015Scale-Free Algorithms for Online Linear Optimizationcited by 30
  11. 2020The Power of Factorial Powers: New Parameter settings for (Stochastic) Optimizationcited by 8
  12. 2020Almost sure convergence rates for Stochastic Gradient Descent and Stochastic Heavy Ballcited by 6
  13. 2020Dual Averaging is Surprisingly Effective for Deep Learning Optimizationcited by 6
  14. 2019Offset Sampling Improves Deep Learning based Accelerated MRI Reconstructions by Exploiting Symmetrycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 7; 2024: 5; 2025: 2; 2026: 1.

27 papers.

Sort:

By area:

  1. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  2. 2024The Road Less Scheduledcited by 158
  3. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
  4. 2021The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactionscited by 31
  5. 2022Deep Symbolic Learning: Discovering Symbols and Rules from Perceptionscited by 20
  6. 2022Noise-aware Physics-informed Machine Learning for Robust PDE Discoverycited by 14
  7. 2023Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Casecited by 13
  8. 2025Shortcuts and Identifiability in Concept-based Models from a Neuro-Symbolic Lenscited by 13
  9. 2022Contrastive Pretraining for Echocardiography Segmentation with Limited Datacited by 12
  10. 2022Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Methodcited by 9
  11. 2022AirTrack: Onboard Deep Learning Framework for Long-Range Aircraft Detection and Trackingcited by 9
  12. 2025Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methodscited by 9
15 more
  1. 2022SD-LayerNet: Semi-supervised retinal layer segmentation in OCT using disentangled representation with anatomical priorscited by 7
  2. 2024Deciphering Hate: Identifying Hateful Memes and Their Targetscited by 7
  3. 2022Wasserstein Adversarial Transformer for Cloud Workload Predictioncited by 4
  4. 2023Skeleton-based Human Action Recognition via Convolutional Neural Networks (CNN)cited by 4
  5. 2023Robust vertebra identification using simultaneous node and edge predicting Graph Neural Networkscited by 3
  6. 2023How Much Context Does My Attention-Based ASR System Need?cited by 3
  7. 2022SSM-Net: feature learning for Music Structure Analysis using a Self-Similarity-Matrix based losscited by 2
  8. 2023Leveraging Cross-Utterance Context For ASR Decodingcited by 1
  9. 2021A theoretical and empirical study of new adaptive algorithms with additional momentum steps and shifted updates for stochastic non-convex optimizationcited by 0
  10. 2022Codec at SemEval-2022 Task 5: Multi-Modal Multi-Transformer Misogynous Meme Classification Frameworkcited by 0
  11. 2023Promoting Exploration in Memory-Augmented Adam using Critical Momentacited by 0
  12. 2024Regularized Adaptive Momentum Dual Averaging with an Efficient Inexact Subproblem Solver for Training Structured Neural Networkcited by 0
  13. 2024Prompt-driven Universal Model for View-Agnostic Echocardiography Analysiscited by 0
  14. 2024Self-Train Before You Transcribecited by 0
  15. 2026Retrieval Augmented Enhanced Dual Co-Attention Framework for Target Aware Multimodal Bengali Hateful Meme Detectioncited by 0