mlsanityData to 7 Oct 2026

Neural networks trained with SGD learn distributions of increasing complexity

arXiv 2211.11567 ↗First posted 21 Nov 2022stat.ML cond-mat.dis-nn cond-mat.stat-mechcited by 50

Not on the map. Why?

Abstract (arXiv)

The ability of deep neural networks to generalise well even when they interpolate their training data has been explained using various "simplicity biases". These theories postulate that neural networks avoid overfitting by first learning simple functions, say a linear classifier, before learning more complex, non-linear functions. Meanwhile, data structure is also recognised as a key ingredient for good generalisation, yet its role in simplicity biases is not yet understood. Here, we show that neural networks trained using stochastic gradient descent initially classify their inputs using lower-order input statistics, like mean and covariance, and exploit higher-order statistics only later during training. We first demonstrate this distributional simplicity bias (DSB) in a solvable model of a neural network trained on synthetic data. We empirically demonstrate DSB in a range of deep convolutional networks and visual transformers trained on CIFAR10, and show that it even holds in networks pre-trained on ImageNet. We discuss the relation of DSB to other simplicity biases and consider its implications for the principle of Gaussian universality in learning.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 58 of its 94 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

58 of 94 references.

References · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2017Improved Training of Wasserstein GANscited by 2,223
  9. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  10. 2018On the Spectral Bias of Neural Networkscited by 1,321
  11. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  12. 2017A Closer Look at Memorization in Deep Networkscited by 936
38 more
  1. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  3. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  7. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  8. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  9. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  10. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  11. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  12. 2017Implicit Regularization in Matrix Factorizationcited by 318
  13. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  14. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  15. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  16. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  17. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  18. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  19. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  20. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  21. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  22. 2020Directional convergence and alignment in deep learningcited by 172
  23. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  24. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  25. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  26. 2021Pre-training without Natural Imagescited by 100
  27. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  28. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  29. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  30. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  31. 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
  32. 2020What Do Neural Networks Learn When Trained With Random Labels?cited by 58
  33. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
  34. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  35. 2018Understanding training and generalization in deep learning by Fourier analysiscited by 43
  36. 2020Distributional Generalization: A New Kind of Generalizationcited by 41
  37. 2020Hierarchical nucleation in deep neural networkscited by 31
  38. 2021Does Pretraining for Summarization Require Knowledge Transfer?cited by 30

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 17; 2025: 11; 2026: 14.

50 papers.

Sort:

By area:

  1. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  2. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 33
  3. 2024Neural Networks Learn Statistics of Increasing Complexitycited by 30
  4. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 23
  5. 2024Position: Understanding LLMs Requires More Than Statistical Generalizationcited by 20
  6. 2023A Mathematical Model for Curriculum Learning for Paritiescited by 19
  7. 2024How transformers learn structured data: insights from hierarchical filteringcited by 18
  8. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  9. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  10. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  11. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  12. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
38 more
  1. 2024High-dimensional learning of narrow neural networkscited by 12
  2. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11
  3. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  4. 2024Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Trainingcited by 8
  5. 2023Linear CNNs Discover the Statistical Structure of the Dataset Using Only the Most Dominant Frequenciescited by 7
  6. 2025Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensionscited by 7
  7. 2026To Use or not to Use Muon: How Simplicity Bias in Optimizers Matterscited by 7
  8. 2024Task Diversity Shortens the ICL Plateaucited by 5
  9. 2025Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architecturescited by 5
  10. 2026A theory of learning data statistics in diffusion models, from easy to hardcited by 5
  11. 2022GPT-Neo for commonsense reasoning -- a theoretical and practical lenscited by 4
  12. 2023Learning from higher-order statistics, efficiently: hypothesis tests, random features, and neural networkscited by 3
  13. 2024Early learning of the optimal constant solution in neural networks and humanscited by 3
  14. 2026Biased Generalization in Diffusion Modelscited by 3
  15. 2024Context-Aware Machine Translation with Source Coreference Explanationcited by 2
  16. 2025Learning Dynamics of RNNs in Closed-Loop Environmentscited by 2
  17. 2025Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lenscited by 2
  18. 2026Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamicscited by 2
  19. 2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
  20. 2024On How Iterative Magnitude Pruning Discovers Local Receptive Fields in Fully Connected Neural Networkscited by 1
  21. 2025Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Taskscited by 1
  22. 2025DNNs, Dataset Statistics, and Correlation Functionscited by 1
  23. 2026It's Not a Lottery, It's a Race: Understanding How Gradient Descent Adapts the Network's Capacity to the Taskcited by 1
  24. 2026Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Trainingcited by 1
  25. 2023Optimal transfer protocol by incremental layer defrostingcited by 0
  26. 2023Loss Dynamics of Temporal Difference Reinforcement Learningcited by 0
  27. 2024Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptroncited by 0
  28. 2025Gaussian Universality for Diffusion Modelscited by 0
  29. 2025Probing Geometry of Next Token Prediction Using Cumulant Expansion of the Softmax Entropycited by 0
  30. 2025Modular connectivity in neural networks emerges from Poisson noise-motivated regularisation, and promotes robustness and compositional generalisationcited by 0
  31. 2026Learning Beyond the Gaussian Data: Learning Dynamics of Neural Networks on an Expressive and Cumulant-Controllable Data Modelcited by 0
  32. 2026A Theory of Saddle Escape in Deep Nonlinear Networkscited by 0
  33. 2026Distributional simplicity bias and effective convexity in Energy Based Modelscited by 0
  34. 2026A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insightscited by 0
  35. 2026Activation Functions, Statistics and Learning of Higher-Order Interactions in Restricted Boltzmann Machinescited by 0
  36. 2026OmniISR: A Unified Framework for Centralized and Federated Learning via Intermediate Supervision and Regularizationcited by 0
  37. 2026Signal-to-Noise Ratio and Sample Size Govern Representational Alignment in Neural Networkscited by 0
  38. 2026Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Biascited by 0