mlsanityData to 7 Oct 2026

Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension

arXiv 2405.14630 ↗First posted 23 May 2024stat.ML cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Bounds on the smallest eigenvalue of the neural tangent kernel (NTK) are a key ingredient in the analysis of neural network optimization and memorization. However, existing results require distributional assumptions on the data and are limited to a high-dimensional setting, where the input dimension d0 scales at least logarithmically in the number of samples n. In this work we remove both of these requirements and instead provide bounds in terms of a measure of the collinearity of the data: notably these bounds hold with high probability even when d0 is held constant versus n. We prove our results through a novel application of the hemisphere transform.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

31 of 48 references.

References · 31

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  8. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  9. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  10. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  11. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  12. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
19 more
  1. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  2. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  3. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  4. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  5. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  6. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  7. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  8. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  9. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  10. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  11. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  12. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  13. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  14. 2016Diverse Neural Network Learns True Target Functionscited by 33
  15. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  16. 2020On Correctness of Automatic Differentiation for Non-Differentiable Functionscited by 20
  17. 2023Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscapecited by 16
  18. 2021Learning curves for Gaussian process regression with power-law priors and targetscited by 14
  19. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 5.

13 papers.

Sort:

  1. 2026Most ReLU Networks Admit Identifiable Parameterscited by 5
  2. 2025Score-based deterministic density samplingcited by 4
  3. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  4. 2024Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projectionscited by 3
  5. 2025Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networkscited by 3
  6. 2025Constraining the outputs of ReLU neural networkscited by 3
  7. 2024A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networkscited by 2
  8. 2025Quantitative convergence of trained single layer neural networks to Gaussian processescited by 1
  9. 2026Tight Worst-Case Bounds for the Smallest Eigenvalue of ReLU NTK Gram Matricescited by 1
  10. 2025Revisiting the Neural Tangent Kernel: the role of large width and depthcited by 0
  11. 2026Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limitcited by 0
  12. 2026The Differential Neural Tangent Kernel and Its Positivitycited by 0
1 more
  1. 2026Dimension-Free Rank Lifting from Random Hyperplane Arrangementscited by 0