mlsanityData to 7 Oct 2026

Implicit Bias of Gradient Descent on Linear Convolutional Networks

arXiv 1806.00468 ↗First posted 1 Jun 2018cs.LG stat.MLcited by 251

Not on the map. Why?

Abstract (arXiv)

We show that gradient descent on full-width linear convolutional networks of depth L converges to a linear predictor related to the ℓ2/L bridge penalty in the frequency domain. This is in contrast to linearly fully connected networks, where gradient descent converges to the hard margin linear support vector machine solution, regardless of depth.

Key facts. Cited by 251 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 48; 2022: 50; 2023: 49; 2024: 38; 2025: 33; 2026: 33.

251 papers.

Sort:

By area:

  1. 2024The Platonic Representation Hypothesiscited by 362
  2. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  3. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  4. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  5. 2022Vision Transformers provably learn spatial structurecited by 92
  6. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  7. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  8. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  9. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  10. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  11. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  12. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
38 more
  1. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  2. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  3. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  4. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  5. 2022Agree to Disagree: Diversity through Disagreement for Better Transferabilitycited by 69
  6. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  7. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  8. 2022Diversify and Disambiguate: Learning From Underspecified Datacited by 55
  9. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  10. 2022Mechanistic Mode Connectivitycited by 50
  11. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  12. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  13. 2021Implicit Regularization in Tensor Factorizationcited by 43
  14. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  15. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  16. 2021SGD Generalizes Better Than GD (And Regularization Doesn't Help)cited by 38
  17. 2021Understanding the role of importance weighting for deep learningcited by 38
  18. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  19. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  20. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  21. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  22. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  23. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 36
  24. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  25. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  26. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  27. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  28. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  29. 2021Fast Margin Maximization via Dual Accelerationcited by 33
  30. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  31. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  32. 2024The Implicit Bias of Adam on Separable Datacited by 32
  33. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  34. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  35. 2023Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppressioncited by 30
  36. 2022Regression as Classification: Influence of Task Formulation on Neural Network Featurescited by 29
  37. 2022Flat minima generalize for low-rank matrix recoverycited by 28
  38. 2021Geometry of Linear Convolutional Networkscited by 27

50 most cited of 251 All 251 (JSON)