Data to 7 Oct 2026

On Lazy Training in Differentiable Programming

arXiv 1812.07956 ↗First posted 19 Dec 2018math.OC cs.LGcited by 671

Not on the map. Why?

Abstract (arXiv)

In a series of recent theoretical works, it was shown that strongly over-parameterized neural networks trained with gradient-based methods could converge exponentially fast to zero training loss, with their parameters hardly varying. In this work, we show that this "lazy training" phenomenon is not specific to over-parameterized neural networks, and is due to a choice of scaling, often implicit, that makes the model behave as its linearization around the initialization, thus yielding a model equivalent to learning with positive-definite kernels. Through a theoretical analysis, we exhibit various situations where this phenomenon arises in non-convex optimization and we provide bounds on the distance between the lazy and linearized optimization paths. Our numerical experiments bring a critical note, as we observe that the performance of commonly used non-linear deep convolutional neural networks in computer vision degrades when trained in the lazy regime. This makes it unlikely that "lazy training" is behind the many successes of neural networks in difficult high dimensional tasks.

Key facts. Cited by 671 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 98; 2022: 120; 2023: 127; 2024: 115; 2025: 97; 2026: 114.

671 papers.

Sort:

By area:

2021Explaining Neural Scaling Lawscited by 353
2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
2022Patching open-vocabulary models by interpolating weightscited by 187
2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
2022Neural Networks can Learn Representations with Gradient Descentcited by 161
2023Birth of a Transformer: A Memory Viewpointcited by 154
2021Dash: Semi-Supervised Learning with Dynamic Thresholdingcited by 147
2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111

38 more

2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 109
2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
2024A Dynamical Model of Neural Scaling Lawscited by 105
2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
2021Priors in Bayesian Deep Learning: A Reviewcited by 101
2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
2023A Spectral Condition for Feature Learningcited by 101
2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
2022Robust Training under Label Noise by Over-parameterizationcited by 93
2022Learning Single-Index Models with Shallow Neural Networkscited by 92
2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
2024Model Collapse Demystified: The Case of Regressioncited by 80
2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
2021Wide Neural Networks Forget Less Catastrophicallycited by 64
2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
2024Strong Model Collapsecited by 62
2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
2021Do Input Gradients Highlight Discriminative Features?cited by 52

50 most cited of 671 All 671 (JSON)