mlsanityData to 7 Oct 2026

Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum

arXiv 2307.14531 ↗First posted 26 Jul 2023cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Wide neural networks are biased towards learning certain functions, influencing both the rate of convergence of gradient descent (GD) and the functions that are reachable with GD in finite training time. As such, there is a great need for methods that can modify this bias according to the task at hand. To that end, we introduce Modified Spectrum Kernels (MSKs), a novel family of constructed kernels that can be used to approximate kernels with desired eigenvalues for which no closed form is known. We leverage the duality between wide neural networks and Neural Tangent Kernels and propose a preconditioned gradient descent method, which alters the trajectory of GD. As a result, this allows for a polynomial and, in some cases, exponential training speedup without changing the final solution. Our method is both computationally efficient and simple to implement.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 49 of its 68 references that are in the map. First posted on arXiv 26 Jul 2023.

Builds on

49 of 68 references.

References · 49

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  7. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 629
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  10. 2017Large Batch Training of Convolutional Networkscited by 529
  11. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  12. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
37 more
  1. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  2. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  3. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  4. 2020Benign overfitting in ridge regressioncited by 152
  5. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  6. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  7. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  8. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  9. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  10. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  11. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  12. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  13. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  14. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  15. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  16. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  17. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  18. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  19. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  20. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 58
  21. 2019Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problemscited by 40
  22. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 36
  23. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  24. 2017Diving into the shallows: a computational perspective on large-scale shallow learningcited by 30
  25. 2020When Does Preconditioning Help or Hurt Generalization?cited by 30
  26. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  27. 2020Generalized Leverage Score Sampling for Neural Networkscited by 26
  28. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  29. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  30. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  31. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 20
  32. 2020Kernel-Based Smoothness Analysis of Residual Networkscited by 19
  33. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  34. 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 15
  35. 2022A Kernel Perspective of Skip Connections in Convolutional Networkscited by 12
  36. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  37. 2022On Kernel Regression with Data-Dependent Kernelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 3; 2026: 2.

9 papers.

  1. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
  2. 2024Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representationscited by 8
  3. 2024Understanding the dynamics of the frequency bias in neural networkscited by 4
  4. 2025Continuous Representation Methods, Theories, and Applications: An Overview and Perspectivescited by 4
  5. 2025Cross-Model Semantics in Representation Learningcited by 2
  6. 2023Changing the Kernel During Training Leads to Double Descent in Kernel Regressioncited by 1
  7. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  8. 2026Pseudo-differential-enhanced physics-informed neural networkscited by 0
  9. 2026Flattening the Connectome Spectrum: A Spectral Filter for FC Induces a Pretraining Target for fMRI Encoderscited by 0