mlsanityData to 14 Oct 2026

Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks

arXiv 2607.07884 ↗First posted 8 Jul 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this short note we consider the gradient descent dynamics of deep scalar linear networks, f(x) = prodl=1L wl x, which enjoy exact time-course solutions for any integer depth. We show that even in this minimal model, the optimal depth-wise learning rate scaling depends on data, whereas data-agnostic scaling rules fail to transfer across depths. Under the data-dependent optimal scaling, the learning dynamics is independent of data and weakly dependent on depth, resulting in a constant linear convergence rate across all depths including infinity. We further show similar data-dependent effects in deep scalar linear networks with residual connections.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 8 Jul 2026.

Builds on

41 of 50 references.

References · 41

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2018Neural Ordinary Differential Equationscited by 3,779
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  4. 2019Deep Equilibrium Modelscited by 615
  5. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  6. 2017Stable Architectures for Deep Neural Networkscited by 341
  7. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  8. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  9. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  10. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  11. 2016Deep Information Propagationcited by 195
  12. 2018A mathematical theory of semantic development in deep neural networkscited by 186
29 more
  1. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  2. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  3. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  4. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  5. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
  6. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
  7. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
  8. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  9. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 76
  10. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  11. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  12. 2024Understanding Optimization in Deep Learning with Central Flowscited by 64
  13. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
  14. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  15. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 39
  16. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  17. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  18. 2018Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networkscited by 22
  19. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  20. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  21. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 19
  22. 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 19
  23. 2022Scaling ResNets in the Large-depth Regimecited by 18
  24. 2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 17
  25. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  26. 2023Depth Dependence of μP Learning Rates in ReLU MLPscited by 9
  27. 2024Three Mechanisms of Feature Learning in a Linear Networkcited by 4
  28. 2025On the inductive bias of infinite-depth ResNets and the bottleneck rankcited by 4
  29. 2026The Impact of Anisotropic Covariance Structure on the Training Dynamics and Generalization Error of Linear Networkscited by 2

Built on by

Not cited yet.