Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Rate
Not on the map. Why?
Abstract (arXiv)
Deep learning has non-convex loss landscape and its optimization dynamics is hard to analyze or control. Nevertheless, the dynamics can be empirically convex-like across various tasks, models, optimizers, hyperparameters, etc. In this work, we examine the applicability of convexity and Lipschitz continuity in deep learning, in order to precisely control the loss dynamics via the learning rate schedules. We illustrate that deep learning quickly becomes weakly convex after a short period of training, and the loss is predicable by an upper bound on the last iterate, which further informs the scaling of optimal learning rate. Through the lens of convexity, we build scaling laws of learning rates and losses that extrapolate as much as 80X across training horizons and 70X across model sizes.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 44 of its 60 references that are in the map. First posted on arXiv 6 Feb 2026.
Builds on
44 of 60 references.
References · 44
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
- 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
32 more
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
- 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
- 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
- 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
- 2017How to Escape Saddle Points Efficientlycited by 284
- 2014The Loss Surfaces of Multilayer Networkscited by 278
- 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
- 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
- 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
- 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
- 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
- 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
- 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
- 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
- 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
- 2018A Walk with SGDcited by 74
- 2024Chinchilla Scaling: A replication attemptcited by 65
- 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
- 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
- 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 41
- 2019SGD Converges to Global Minimum in Deep Learning via Star-convex Pathcited by 39
- 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
- 2024Scaling Optimal LR Across Token Horizonscited by 35
- 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 31
- 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Modelscited by 24
- 2016An empirical analysis of the optimization of deep network loss surfacescited by 21
- 2019Convex Formulation of Overparameterized Deep Neural Networkscited by 16
- 2024Gradient descent with generalized Newton's methodcited by 9
- 2019Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representationscited by 7
Built on by
Citing papers per year: 2026: 3.
3 papers.