mlsanityData to 7 Oct 2026

How You Start Matters for Generalization

arXiv 2206.08558 ↗First posted 17 Jun 2022cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Characterizing the remarkable generalization properties of over-parameterized neural networks remains an open problem. In this paper, we promote a shift of focus towards initialization rather than neural architecture or (stochastic) gradient descent to explain this implicit regularization. Through a Fourier lens, we derive a general result for the spectral bias of neural networks and show that the generalization of neural networks is heavily tied to their initialization. Further, we empirically solidify the developed theoretical insights using practical, deep networks. Finally, we make a case against the controversial flat-minima conjecture and show that Fourier analysis grants a more reliable framework for understanding the generalization of neural networks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 66 of its 97 references that are in the map. First posted on arXiv 17 Jun 2022.

Builds on

66 of 97 references.

References · 66

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  9. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  10. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  11. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  12. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
38 more
  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2018On the Spectral Bias of Neural Networkscited by 1,321
  3. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  4. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  5. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  6. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  7. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  8. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  9. 2018On Lazy Training in Differentiable Programmingcited by 671
  10. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  11. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  12. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  13. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  14. 2017Implicit Regularization in Matrix Factorizationcited by 318
  15. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  16. 2017Three Factors Influencing Minima in SGDcited by 270
  17. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  18. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  19. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  20. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  21. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  22. 2019Texture Fields: Learning Texture Representations in Function Spacecited by 210
  23. 2022Patches Are All You Need?cited by 201
  24. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  25. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  26. 2018Training behavior of deep neural network in frequency domaincited by 174
  27. 2020DeRF: Decomposed Radiance Fieldscited by 163
  28. 2021NeRF--: Neural Radiance Fields Without Known Camera Parameterscited by 155
  29. 2019NASA: Neural Articulated Shape Approximationcited by 149
  30. 2021Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPscited by 134
  31. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  32. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  33. 2021A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representationcited by 113
  34. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  35. 2021Neural-GIF: Neural Generalized Implicit Functions for Animating People in Clothingcited by 102
  36. 2021NeuTex: Neural Texture Mapping for Volumetric Neural Renderingcited by 98
  37. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  38. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 2.

5 papers.

  1. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  2. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  3. 2023BLiRF: Bandlimited Radiance Fields for Dynamic Scene Modelingcited by 9
  4. 2025Enhancing Transformers Through Conditioned Embedded Tokenscited by 3
  5. 2025Teleportation With Null Space Gradient Projection for Optimization Accelerationcited by 2