mlsanityData to 7 Oct 2026

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks

arXiv 2006.14599 ↗First posted 25 Jun 2020cs.LG cs.NE stat.MLcited by 58

Not on the map. Why?

Abstract (arXiv)

Modern neural networks are often regarded as complex black-box functions whose behavior is difficult to understand owing to their nonlinear dependence on the data and the nonconvexity in their loss landscapes. In this work, we show that these common perceptions can be completely false in the early phase of learning. In particular, we formally prove that, for a class of well-behaved input distributions, the early-time learning dynamics of a two-layer fully-connected neural network can be mimicked by training a simple linear model on the inputs. We additionally argue that this surprising simplicity can persist in networks with more layers and with convolutional architecture, which we verify empirically. Key to our analysis is to bound the spectral norm of the difference between the Neural Tangent Kernel (NTK) at initialization and an affine transform of the data kernel; however, unlike many previous results utilizing the NTK, we do not require the network to have disproportionately large width, and the network is allowed to escape the kernel regime later in training.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 21; 2023: 10; 2024: 7; 2025: 6; 2026: 4.

58 papers.

Sort:

By area:

  1. 2022Toy Models of Superpositioncited by 928
  2. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  3. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  4. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 86
  5. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  6. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  7. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  8. 2022Mechanistic Mode Connectivitycited by 50
  9. 2022Investigating Why Contrastive Learning Benefits Robustness Against Label Noisecited by 46
  10. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  11. 2023Identifying Spurious Biases Early in Training through the Lens of Simplicity Biascited by 44
  12. 2023Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networkscited by 42
38 more
  1. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  2. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  3. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 31
  4. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  5. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  6. 2022On the Activation Function Dependence of the Spectral Bias of Neural Networkscited by 25
  7. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  8. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  9. 2021RATT: Leveraging Unlabeled Data to Guarantee Generalizationcited by 20
  10. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  11. 2021Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noisecited by 17
  12. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  13. 2023Data Distillation Can Be Like Vodka: Distilling More Times For Better Qualitycited by 17
  14. 2022Fighting Fire with Fire: Avoiding DNN Shortcuts through Primingcited by 15
  15. 2023Latent State Models of Training Dynamicscited by 15
  16. 2022Critical Learning Periods for Multisensory Integration in Deep Networkscited by 14
  17. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  18. 2022Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomialscited by 13
  19. 2023Beyond Distribution Shift: Spurious Features Through the Lens of Training Dynamicscited by 11
  20. 2021Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networkscited by 10
  21. 2022DeepMed: Semiparametric Causal Mediation Analysis with Debiased Deep Learningcited by 10
  22. 2022Which Shortcut Solution Do Question Answering Models Prefer to Learn?cited by 9
  23. 2022Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensionscited by 8
  24. 2022When Expressivity Meets Trainability: Fewer than n Neurons Can Workcited by 8
  25. 2023Understanding the Initial Condensation of Convolutional Neural Networkscited by 7
  26. 2024Predicting the Susceptibility of Examples to Catastrophic Forgettingcited by 7
  27. 2026To Use or not to Use Muon: How Simplicity Bias in Optimizers Matterscited by 7
  28. 2022Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithmcited by 6
  29. 2022Overparameterized random feature regression with nearly orthogonal datacited by 6
  30. 2021Universal scaling laws in the gradient descent training of neural networkscited by 4
  31. 2024GenEFT: Understanding Statics and Dynamics of Model Generalization via Effective Theorycited by 4
  32. 2025On the Role of Label Noise in the Feature Learning Processcited by 4
  33. 2021FEAR: A Simple Lightweight Method to Rank Architecturescited by 3
  34. 2022Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regressioncited by 3
  35. 2024Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalizationcited by 3
  36. 2024Early learning of the optimal constant solution in neural networks and humanscited by 3
  37. 2025Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitationcited by 3
  38. 2024UnLearning from Experience to Avoid Spurious Correlationscited by 2

50 most cited of 58 All 58 (JSON)