mlsanityData to 7 Oct 2026

Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy

arXiv 2007.06738 ↗First posted 13 Jul 2020cs.LG stat.MLcited by 70

Not on the map. Why?

Abstract (arXiv)

We provide a detailed asymptotic study of gradient flow trajectories and their implicit optimization bias when minimizing the exponential loss over "diagonal linear networks". This is the simplest model displaying a transition between "kernel" and non-kernel ("rich" or "active") regimes. We show how the transition is controlled by the relationship between the initialization scale and how accurately we minimize the training loss. Our results indicate that some limit behaviors of gradient descent only kick in at ridiculous training accuracies (well beyond 10-100). Moreover, the implicit bias at reasonable initialization scales and training accuracies is more complex and not captured by these limits.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 16; 2023: 13; 2024: 13; 2025: 5; 2026: 3.

70 papers.

Sort:

  1. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  2. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  3. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  4. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  5. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  6. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  7. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  8. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  9. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  10. 2023Transformers as Support Vector Machinescited by 67
  11. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  12. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
38 more
  1. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  2. 2023Transformers learn through gradual rank increasecited by 52
  3. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  4. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  5. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  6. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  7. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  8. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 34
  9. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  10. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  11. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  12. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  13. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  14. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  15. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  16. 2023Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Lossescited by 24
  17. 2021Understanding Deflation Process in Over-parametrized Tensor Decompositioncited by 20
  18. 2021Towards Understanding Learning in Neural Networks with Linear Teacherscited by 19
  19. 2024How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teacherscited by 18
  20. 2021Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noisecited by 17
  21. 2021A Theoretical Analysis of Fine-tuning with Linear Teacherscited by 17
  22. 2022Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measurescited by 17
  23. 2024Implicit Regularization of Gradient Flow on One-Layer Softmax Attentioncited by 16
  24. 2021Optimal Gradient-based Algorithms for Non-concave Bandit Optimizationcited by 15
  25. 2021Implicit Regularization in Matrix Sensing via Mirror Descentcited by 11
  26. 2024Towards Robust Out-of-Distribution Generalization Bounds via Sharpnesscited by 10
  27. 2021Going Beyond Linear RL: Sample Efficient Neural Function Approximationcited by 9
  28. 2021Parallel Deep Neural Networks Have Zero Duality Gapcited by 9
  29. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  30. 2021Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samplescited by 8
  31. 2022Thinking Outside the Ball: Optimal Learning with Gradient Descent for Generalized Linear Stochastic Convex Optimizationcited by 7
  32. 2023The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Modelscited by 7
  33. 2024Implicit Bias of Mirror Flow on Separable Datacited by 7
  34. 2024Landscaping Linear Mode Connectivitycited by 7
  35. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 6
  36. 2022How You Start Matters for Generalizationcited by 5
  37. 2022Adaptive and Implicit Regularization for Matrix Completioncited by 5
  38. 2022Magnitude and Angle Dynamics in Training Single ReLU Neuronscited by 5

50 most cited of 70 All 70 (JSON)