mlsanityData to 7 Oct 2026

Understanding training and generalization in deep learning by Fourier analysis

arXiv 1808.04295 ↗First posted 13 Aug 2018cs.LG cs.AI math.OCcited by 43

Not on the map. Why?

Abstract (arXiv)

Background: It is still an open research area to theoretically understand why Deep Neural Networks (DNNs)---equipped with many more parameters than training data and trained by (stochastic) gradient-based methods---often achieve remarkably low generalization error. Contribution: We study DNN training by Fourier analysis. Our theoretical framework explains: i) DNN with (stochastic) gradient-based methods often endows low-frequency components of the target function with a higher priority during the training; ii) Small initialization leads to good generalization ability of DNN while preserving the DNN's ability to fit any function. These results are further confirmed by experiments of DNNs fitting the following datasets, that is, natural images, one-dimensional functions and MNIST dataset.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 9; 2023: 2; 2024: 14; 2025: 7; 2026: 3.

43 papers.

Sort:

  1. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  2. 2021Geometry-Consistent Neural Shape Representation with Implicit Displacement Fieldscited by 60
  3. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  4. 2023Adaptive Frequency Filters As Efficient Global Token Mixerscited by 45
  5. 2021Implicit Regularization in Tensor Factorizationcited by 43
  6. 2021Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networkscited by 43
  7. 2021Discovering and Explaining the Representation Bottleneck of DNNscited by 42
  8. 2022Deep Frequency Filtering for Domain Generalizationcited by 41
  9. 2024Improved Implicit Neural Representation with Fourier Reparameterized Trainingcited by 36
  10. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
  11. 2022On the Activation Function Dependence of the Spectral Bias of Neural Networkscited by 25
  12. 2024SpeGCL: Self-supervised Graph Spectrum Contrastive Learning without Positive Samplescited by 20
31 more
  1. 2024MLP Can Be A Good Transformer Learnercited by 16
  2. 2024FreSh: Frequency Shifting for Accelerated Neural Representation Learningcited by 14
  3. 2024Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learningcited by 14
  4. 2021Exploring The Effect of High-frequency Components in GANs Trainingcited by 10
  5. 2024Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Trainingcited by 8
  6. 2024Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representationscited by 8
  7. 2023Linear CNNs Discover the Statistical Structure of the Dataset Using Only the Most Dominant Frequenciescited by 7
  8. 2024Exploring the Low-Pass Filtering Behavior in Image Super-Resolutioncited by 7
  9. 2022Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithmcited by 6
  10. 2022How You Start Matters for Generalizationcited by 5
  11. 2024Frequency-adaptive Multi-scale Deep Neural Networkscited by 5
  12. 2025Spectral Bias in Variational Quantum Machine Learningcited by 5
  13. 2021Trap of Feature Diversity in the Learning of MLPscited by 4
  14. 2024Understanding the dynamics of the frequency bias in neural networkscited by 4
  15. 2024Implicit Neural Representations with Fourier Kolmogorov-Arnold Networkscited by 4
  16. 2025PDEfuncta: Spectrally-Aware Neural Representation for PDE Solution Modelingcited by 4
  17. 2021An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Networkcited by 3
  18. 2022Towards Theoretical Analysis of Transformation Complexity of ReLU DNNscited by 2
  19. 2024Towards the Spectral bias Alleviation by Normalizations in Coordinate Networkscited by 2
  20. 2022Win the Lottery Ticket via Fourier Analysis: Frequencies Guided Network Pruningcited by 1
  21. 2025From Tables to Signals: Revealing Spectral Adaptivity in TabPFNcited by 1
  22. 2025The Interaction Bottleneck of Deep Neural Networks: Discovery, Proof, and Modulationcited by 1
  23. 2022Explanation-based Counterfactual Retraining(XCR): A Calibration Method for Black-box Modelscited by 0
  24. 2024SL2A-INR: Single-Layer Learnable Activation for Implicit Neural Representationcited by 0
  25. 2024Un-mixing Test-time Adaptation under Heterogeneous Data Streamscited by 0
  26. 2025LFR-PINO: A Layered Fourier Reduced Physics-Informed Neural Operator for Parametric PDEscited by 0
  27. 2025FLAIR: Frequency- and Locality-Aware Implicit Neural Representationscited by 0
  28. 2025Frequency-adaptive tensor neural networks for high-dimensional multi-scale problemscited by 0
  29. 2026Pseudo-differential-enhanced physics-informed neural networkscited by 0
  30. 2026Spectral Energy Centroid: a Metric for Improving Performance and Analyzing Spectral Bias in Implicit Neural Representationscited by 0
  31. 2026See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learningcited by 0