mlsanityData to 7 Oct 2026

The Road Less Scheduled

arXiv 2405.15682 ↗First posted 24 May 2024cs.LG cs.AI math.OCcited by 158

Not on the map. Why?

Abstract (arXiv)

Existing learning rate schedules that do not require specification of the optimization stopping step T are greatly out-performed by learning rate schedules that depend on T. We propose an approach that avoids the need for this stopping time by eschewing the use of schedules entirely, while exhibiting state-of-the-art performance compared to schedules across a wide family of problems ranging from convex problems to large-scale deep learning problems. Our Schedule-Free approach introduces no additional hyper-parameters over standard optimizers with momentum. Our method is a direct consequence of a new theory we develop that unifies scheduling and iterate averaging. An open source implementation of our method is available at https://github.com/facebookresearch/schedule_free. Schedule-Free AdamW is the core algorithm behind our winning entry to the MLCommons 2024 AlgoPerf Algorithmic Efficiency Challenge Self-Tuning track.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Builds on 35 of its 71 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

35 of 71 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2016Wide Residual Networkscited by 3,057
  9. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  11. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  12. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
23 more
  1. 2019On the Convergence of Adam and Beyondcited by 834
  2. 2019Introduction to Online Convex Optimizationcited by 793
  3. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  4. 2018fastMRI: An Open Dataset and Benchmarks for Accelerated MRIcited by 464
  5. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 407
  6. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  7. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  8. 2020End-to-End Variational Networks for Accelerated MRI Reconstructioncited by 201
  9. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  10. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  11. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 110
  12. 2019UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimizationcited by 56
  13. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  14. 2023Benchmarking Neural Network Training Algorithmscited by 46
  15. 2021Rethinking "Batch" in BatchNormcited by 44
  16. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 36
  17. 2021Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimizationcited by 27
  18. 2023Exact convergence rate of the last iterate in subgradient methodscited by 24
  19. 2017A Modular Analysis of Adaptive (Non-)Convex Optimization: Optimism, Composite Objectives, and Variational Boundscited by 23
  20. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 20
  21. 2020Momentum via Primal Averaging: Theoretical Insights and Learning Rate Schedules for Non-Convex Optimizationcited by 19
  22. 2020The Power of Factorial Powers: New Parameter settings for (Stochastic) Optimizationcited by 8
  23. 2022Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Ratescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 72; 2026: 62.

158 papers.

Sort:

By area:

  1. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  2. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  3. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  4. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 97
  5. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  6. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  7. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  8. 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 43
  9. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 42
  10. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
  11. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 34
  12. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
38 more
  1. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 21
  2. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 21
  3. 20244-bit Shampoo for Memory-Efficient Network Trainingcited by 16
  4. 2025MedConv: Convolutions Beat Transformers on Long-Tailed Bone Density Predictioncited by 16
  5. 2024JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resourcescited by 15
  6. 2025A Stable Whitening Optimizer for Efficient Neural Network Trainingcited by 15
  7. 2024General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimizationcited by 14
  8. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 13
  9. 2025Simulating Three-dimensional Turbulence with Physics-informed Neural Networkscited by 13
  10. 2024Gradient Methods with Online Scalingcited by 11
  11. 2025A Probabilistic Neuro-symbolic Layer for Algebraic Constraint Satisfactioncited by 11
  12. 2026Relational In-Context Learning via Synthetic Pre-training with Structural Priorcited by 11
  13. 2024An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizescited by 10
  14. 2026MultiModalPFN: Extending Prior-Data Fitted Networks for Multimodal Tabular Learningcited by 9
  15. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
  16. 2025Taming Transformer Without Using Learning Rate Warmupcited by 8
  17. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 8
  18. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 8
  19. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 8
  20. 2025Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-trainingcited by 7
  21. 2025GAITGen: Disentangled Motion-Pathology Impaired Gait Generative Model -- Bringing Motion Generation to the Clinical Domaincited by 7
  22. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 7
  23. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 7
  24. 2025SSDD: Single-Step Diffusion Decoder for Efficient Image Tokenizationcited by 7
  25. 2025Light-Weight Diffusion Multiplier and Uncertainty Quantification for Fourier Neural Operatorscited by 6
  26. 2025Unbiased Gradient Low-Rank Projectioncited by 6
  27. 2026TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizerscited by 6
  28. 2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 6
  29. 2026A Defense of the Quadratic Modelcited by 6
  30. 2025Provable and Practical Online Learning Rate Adaptation with Hypergradient Descentcited by 5
  31. 2025S-EO: A Large-Scale Dataset for Geometry-Aware Shadow Detection in Remote Sensing Applicationscited by 5
  32. 2025Optimization Benchmark for Diffusion Models on Dynamical Systemscited by 5
  33. 2025Smoothing DiLoCo with Primal Averaging for Faster Training of LLMscited by 5
  34. 2026What Matters for Scalable and Robust Learning in End-to-End Driving Planners?cited by 5
  35. 2024Optimizing Brain Tumor Segmentation with MedNeXt: BraTS 2024 SSA and Pediatricscited by 4
  36. 2025Improving Diffusion-based Inverse Algorithms under Few-Step Constraint via Learnable Linear Extrapolationcited by 4
  37. 2025Cohort-Based Active Modality Acquisitioncited by 4
  38. 2025Scaling Laws and Symmetry, Evidence from Neural Force Fieldscited by 4

50 most cited of 158 All 158 (JSON)