mlsanityData to 7 Oct 2026

Prodigy: An Expeditiously Adaptive Parameter-Free Learner

arXiv 2306.06101 ↗First posted 9 Jun 2023cs.LG cs.AI math.OCcited by 129

Not on the map. Why?

Cited by 24 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We consider the problem of estimating the learning rate in adaptive methods, such as AdaGrad and Adam. We propose Prodigy, an algorithm that provably estimates the distance to the solution D, which is needed to set the learning rate optimally. At its core, Prodigy is a modification of the D-Adaptation method for learning-rate-free learning. It improves upon the convergence rate of D-Adaptation by a factor of O(√log(D/d0)), where d0 is the initial estimate of D. We test Prodigy on 12 common logistic-regression benchmark datasets, VGG11 and ResNet-50 training on CIFAR10, ViT training on Imagenet, LSTM training on IWSLT14, DLRM training on Criteo dataset, VarNet on Knee MRI dataset, as well as RoBERTa and GPT transformer training on BookWiki. Our experimental results show that our approach consistently outperforms D-Adaptation and reaches test accuracy values close to that of hand-tuned Adam.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

28 of 36 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2019Advances and Open Problems in Federated Learningcited by 3,021
  8. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  9. 2018fastMRI: An Open Dataset and Benchmarks for Accelerated MRIcited by 464
  10. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  11. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  12. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
16 more
  1. 2018Black-Box Reductions for Parameter-free Online Learning in Banach Spacescited by 124
  2. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  3. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  4. 2019Adaptive Gradient Descent without Descentcited by 79
  5. 2021Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharingcited by 58
  6. 2019UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimizationcited by 56
  7. 2019Revisiting the Polyak step sizecited by 55
  8. 2018Online Adaptive Methods, Universality and Accelerationcited by 51
  9. 2022Making SGD Parameter-Freecited by 50
  10. 2014Unconstrained Online Linear Learning in Hilbert Spaces: Minimax Algorithms and Normal Approximationscited by 46
  11. 2022Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solutioncited by 40
  12. 2021Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimizationcited by 27
  13. 2021Parameter-free Stochastic Optimization of Variationally Coherent Functionscited by 27
  14. 2022PDE-Based Optimal Strategy for Unconstrained Online Learningcited by 27
  15. 2021Stochastic Polyak Stepsize with a Moving Targetcited by 19
  16. 2023Adaptive proximal algorithms for convex optimization under local Lipschitz continuity of the gradientcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 29; 2025: 57; 2026: 40.

129 papers.

Sort:

By area:

  1. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 334
  2. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  3. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
  4. 2024Old Optimizer, New Norm: An Anthologycited by 173
  5. 2024Scalable Optimization in the Modular Normcited by 64
  6. 2025Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Taskscited by 62
  7. 2025Insert Anything: Image Insertion via In-Context Editing in DiTcited by 61
  8. 2025OminiControl2: Efficient Conditioning for Diffusion Transformerscited by 56
  9. 2025From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuningcited by 53
  10. 2025FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editingcited by 44
  11. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  12. 2025Accelerating Neural Network Training: An Analysis of the AlgoPerf Competitioncited by 30
38 more
  1. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 29
  2. 2026Controlled LLM Training on Spectral Spherecited by 28
  3. 2025Generating Multi-Image Synthetic Data for Text-to-Image Customizationcited by 25
  4. 2024Kvasir-VQA: A Text-Image Pair GI Tract Datasetcited by 23
  5. 2025IntrinsiX: High-Quality PBR Generation using Image Priorscited by 18
  6. 2025ContextGen: Contextual Layout Anchoring for Identity-Consistent Multi-Instance Generationcited by 17
  7. 2025Personalized Vision via Visual In-Context Learningcited by 14
  8. 2025DreamFuse: Adaptive Image Fusion with Diffusion Transformercited by 13
  9. 2024How Free is Parameter-Free Stochastic Optimization?cited by 12
  10. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  11. 2024The Price of Adaptivity in Stochastic Convex Optimizationcited by 11
  12. 2024Fully Unconstrained Online Learningcited by 10
  13. 2024An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizescited by 10
  14. 2025Bokeh Diffusion: Defocus Blur Control in Text-to-Image Diffusion Modelscited by 10
  15. 2025Image Editing As Programs with Diffusion Modelscited by 10
  16. 2024Gradient descent with generalized Newton's methodcited by 9
  17. 2025InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attentioncited by 9
  18. 2025BideDPO: Conditional Image Generation with Simultaneous Text and Condition Alignmentcited by 9
  19. 2024Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGradcited by 8
  20. 2024PALM: Pushing Adaptive Learning Rate Mechanisms for Continual Test-Time Adaptationcited by 8
  21. 2024Neural Pfaffians: Solving Many Many-Electron Schr"odinger Equationscited by 8
  22. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  23. 2025CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-stepcited by 8
  24. 2024Lift Your Molecules: Molecular Graph Generation in Latent Euclidean Spacecited by 7
  25. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  26. 2026LuxRemix: Lighting Decomposition and Remixing for Indoor Scenescited by 7
  27. 2024No More Adam: Learning Rate Scaling at Initialization is All You Needcited by 6
  28. 2025MC-VTON: Minimal Control Virtual Try-On Diffusion Transformercited by 6
  29. 2025UTDesign: A Unified Framework for Stylized Text Editing and Generation in Graphic Design Imagescited by 6
  30. 2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 6
  31. 2023Non-Uniform Smoothness for Gradient Descentcited by 5
  32. 2023SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithmscited by 5
  33. 2024MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameterscited by 5
  34. 2025Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillationcited by 5
  35. 2025SIGMA-GEN: Structure and Identity Guided Multi-subject Assembly for Image Generationcited by 5
  36. 2025Optimization Benchmark for Diffusion Models on Dynamical Systemscited by 5
  37. 2025Nesterov Finds GRAAL: Optimal and Adaptive Gradient Method for Convex Optimizationcited by 4
  38. 2025Mixture of Global and Local Experts with Diffusion Transformer for Controllable Face Generationcited by 4

50 most cited of 129 All 129 (JSON)