mlsanityData to 7 Oct 2026

A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights

arXiv 2605.16913 ↗First posted 16 May 2026stat.ML cond-mat.dis-nn cond-mat.stat-mechcited by 0

Not on the map. Why?

Abstract (arXiv)

Neural networks trained with gradient-based methods exhibit a strong simplicity bias: they learn simpler statistical features of their data before moving to more complex features. Previous analyses of this phenomenon have largely focused on settings with (quasi-)isotropic inputs. In this work, we study the simplicity bias from a Fourier perspective, which allows us to include two key features of natural images in the analysis: approximate translation-invariance and power-law spectra. We first show experimentally that simple neural networks trained on image classification tasks first rely on amplitude information -- related to pair-wise correlations between pixels -- before exploiting phase information, which encodes edges and higher-order correlations. In view of this, we introduce a synthetic data model for translation-invariant inputs that allows precise control over amplitudes and phases while remaining tractable. We rigorously establish that for isotropic and high-dimensional inputs, classification based on phase information alone is a genuinely hard task: online stochastic gradient descent (SGD) cannot distinguish the structured inputs from noise within n ll N3 steps, but needs at least n gg N3 log2N steps. In contrast, we show both experimentally and theoretically that power-law spectra can dramatically accelerate the speed of learning phase information, even if the spectra do not help with classification. Simulations with two-layer networks trained on textures and with deep convolutional networks on ImageNet and CIFAR100 confirm this non-trivial interaction between amplitudes and phases, providing mechanistic insights into how deep neural networks can learn natural image distributions efficiently.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 68 references that are in the map. First posted on arXiv 16 May 2026.

Builds on

39 of 68 references.

References · 39

  1. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  2. 2018On the Spectral Bias of Neural Networkscited by 1,321
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2019Benign Overfitting in Linear Regressioncited by 544
  5. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  6. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  7. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  8. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  9. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  10. 2019Notes on Computational Hardness of Hypothesis Testing: Predictions using the Low-Degree Likelihood Ratiocited by 82
  11. 2014A statistical model for tensor PCAcited by 72
  12. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
27 more
  1. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  2. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 66
  3. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  4. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 60
  5. 2023Learning time-scales in two-layers neural networkscited by 57
  6. 2022Dimension free ridge regressioncited by 50
  7. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  8. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
  9. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  10. 2024Neural Networks Learn Statistics of Increasing Complexitycited by 30
  11. 2022Data-driven emergence of convolutional structure in neural networkscited by 29
  12. 2025Emergence and scaling laws in SGD learning of shallow neural networkscited by 29
  13. 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trainedcited by 24
  14. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 23
  15. 2025Learning quadratic neural networks in high dimensions: SGD dynamics and scaling lawscited by 23
  16. 2024How transformers learn structured data: insights from hierarchical filteringcited by 18
  17. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 18
  18. 2023Spectral alignment of stochastic gradient descent for high-dimensional classification taskscited by 13
  19. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
  20. 2025Kernel ridge regression under power-law data: spectrum and generalizationcited by 8
  21. 2023Linear CNNs Discover the Statistical Structure of the Dataset Using Only the Most Dominant Frequenciescited by 7
  22. 2025Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensionscited by 7
  23. 2026A solvable high-dimensional model where nonlinear autoencoders learn structure invisible to PCA while test loss misaligns with generalizationcited by 7
  24. 2024Stochastic gradient descent in high dimensions for multi-spiked tensor PCAcited by 6
  25. 2024Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depthcited by 4
  26. 2025Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Datacited by 3
  27. 2026Perceptual misalignment of texture representations in convolutional neural networkscited by 2

Built on by

Not cited yet.