mlsanityData to 7 Oct 2026

Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data

arXiv 2210.07082 ↗First posted 13 Oct 2022cs.LG stat.MLcited by 46

Not on the map. Why?

Abstract (arXiv)

The implicit biases of gradient-based optimization algorithms are conjectured to be a major factor in the success of modern deep learning. In this work, we investigate the implicit bias of gradient flow and gradient descent in two-layer fully-connected neural networks with leaky ReLU activations when the training data are nearly-orthogonal, a common property of high-dimensional data. For gradient flow, we leverage recent work on the implicit bias for homogeneous neural networks to show that asymptotically, gradient flow produces a neural network with rank at most two. Moreover, this network is an ℓ2-max-margin solution (in parameter space), and has a linear decision boundary that corresponds to an approximate-max-margin linear predictor. For gradient descent, provided the random initialization variance is small enough, we show that a single step of gradient descent suffices to drastically reduce the rank of the network, and that the rank remains small throughout training. We provide experiments which suggest that a small initialization scale is important for finding low-rank neural networks with gradient descent.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

32 of 42 references.

References · 32

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  5. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  6. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  7. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  8. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  9. 2020Directional convergence and alignment in deep learningcited by 172
  10. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  11. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  12. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
20 more
  1. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  2. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  3. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  4. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  5. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  6. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  7. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  8. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  9. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  10. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  11. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  12. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 33
  13. 2022Gradient Methods Provably Converge to Non-Robust Networkscited by 27
  14. 2018Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalizationcited by 21
  15. 2021Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descentcited by 21
  16. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  17. 2021Towards Understanding Learning in Neural Networks with Linear Teacherscited by 19
  18. 2022Data Augmentation as Feature Manipulationcited by 18
  19. 2021Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noisecited by 17
  20. 2019Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networkscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 19; 2024: 13; 2025: 10; 2026: 3.

46 papers.

Sort:

  1. 2023Transformers learn through gradual rank increasecited by 52
  2. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  3. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  4. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
  5. 2024The Implicit Bias of Adam on Separable Datacited by 32
  6. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  7. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  8. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  9. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  10. 2023Simplicity Bias in 1-Hidden Layer Neural Networkscited by 26
  11. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  12. 2023Generalization and Stability of Interpolating Neural Networks with Minimal Widthcited by 21
34 more
  1. 2023Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Datacited by 18
  2. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  3. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
  4. 2024Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Contextcited by 14
  5. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  6. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  7. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 13
  8. 2024Non-asymptotic Convergence of Training Transformers for Next-token Predictioncited by 13
  9. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 12
  10. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11
  11. 2025How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Biascited by 10
  12. 2024Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selectioncited by 9
  13. 2022Overparameterized random feature regression with nearly orthogonal datacited by 6
  14. 2024Can Implicit Bias Imply Adversarial Robustness?cited by 5
  15. 2023Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regressioncited by 4
  16. 2024Provable Benefit of Cutout and CutMix for Feature Learningcited by 4
  17. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 4
  18. 2025Low Rank Gradients and Where to Find Themcited by 4
  19. 2023ESMC: Entire Space Multi-Task Model for Post-Click Conversion Rate via Parameter Constraintcited by 3
  20. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 3
  21. 2023The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learningcited by 2
  22. 2023A Test-Time Learning Approach to Reparameterize the Geophysical Inverse Problem with a Convolutional Neural Networkcited by 2
  23. 2024Theoretical Understanding of Learning from Adversarial Perturbationscited by 2
  24. 2025Convergence of Shallow ReLU Networks on Weakly Interacting Datacited by 2
  25. 2025Towards Understanding the Benefits of Neural Network Parameterizations in Geophysical Inversions: A Study With Neural Fieldscited by 2
  26. 2023Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networkscited by 1
  27. 2025Wormhole Dynamics in Deep Neural Networkscited by 1
  28. 2023Frequency maps reveal the correlation between Adversarial Attacks and Implicit Biascited by 0
  29. 2023Sparse Mean Estimation in Adversarial Settings via Incremental Learningcited by 0
  30. 2025Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networkscited by 0
  31. 2025Directional Convergence, Benign Overfitting of Gradient Descent in leaky ReLU two-layer Neural Networkscited by 0
  32. 2026Emergent Low-Rank Training Dynamics in MLPs with Smooth Activationscited by 0
  33. 2026Sufficient Conditions for Stability of Minimum-Norm Interpolating Deep ReLU Networkscited by 0
  34. 2026On the Rate of Convergence of GD in Non-linear Neural Networks: An Adversarial Robustness Perspectivecited by 0