mlsanityData to 7 Oct 2026

Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks

arXiv 2012.11654 ↗First posted 21 Dec 2020stat.ML cs.LGcited by 87

Not on the map. Why?

Abstract (arXiv)

A recent line of work has analyzed the theoretical properties of deep neural networks via the Neural Tangent Kernel (NTK). In particular, the smallest eigenvalue of the NTK has been related to the memorization capacity, the global convergence of gradient descent algorithms and the generalization of deep nets. However, existing results either provide bounds in the two-layer setting or assume that the spectrum of the NTK matrices is bounded away from 0 for multi-layer networks. In this paper, we provide tight bounds on the smallest eigenvalue of NTK matrices for deep ReLU nets, both in the limiting case of infinite widths and for finite widths. In the finite-width setting, the network architectures we consider are fairly general: we require the existence of a wide layer with roughly order of N neurons, N being the number of data samples; and the scaling of the remaining layer widths is arbitrary (up to logarithmic factors). To obtain our results, we analyze various quantities of independent interest: we give lower bounds on the smallest singular value of hidden feature matrices, and upper bounds on the Lipschitz constant of input-output feature maps.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 19; 2023: 20; 2024: 18; 2025: 14; 2026: 6.

87 papers.

Sort:

By area:

  1. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  2. 2024Scaling Laws for Precisioncited by 92
  3. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  4. 2024Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalizationcited by 39
  5. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  6. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  7. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  8. 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 30
  9. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
  10. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  11. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  12. 2022Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)cited by 18
38 more
  1. 2024Reparameterization invariance in approximate Bayesian inferencecited by 17
  2. 2021On the Expected Complexity of Maxout Networkscited by 15
  3. 2023Sample Complexity Bounds for Score-Matching: Causal Discovery and Generative Modelingcited by 15
  4. 2021Convergence of Deep ReLU Networkscited by 14
  5. 2022Generalization Properties of NAS under Activation and Skip Connection Searchcited by 14
  6. 2023Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernelscited by 13
  7. 2023On the Convergence of Encoder-only Shallow Transformerscited by 13
  8. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  9. 2024The Positivity of the Neural Tangent Kernelcited by 12
  10. 2022Phenomenology of Double Descent in Finite-Width Neural Networkscited by 11
  11. 2023How Spurious Features Are Memorized: Precise Analysis for Random and NTK Featurescited by 11
  12. 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10
  13. 2022On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networkscited by 9
  14. 2022Lower and Upper Bounds for Numbers of Linear Regions of Graph Convolutional Networkscited by 9
  15. 2023Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrumcited by 9
  16. 2023Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networkscited by 9
  17. 2025Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterizationcited by 9
  18. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  19. 2023Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approachcited by 8
  20. 2021TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversioncited by 7
  21. 2021On the Role of Optimization in Double Descent: A Least Squares Studycited by 7
  22. 2022Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Studycited by 7
  23. 2022Restricted Strong Convexity of Deep Learning Models with Smooth Activationscited by 7
  24. 2022On skip connections and normalisation layers in deep optimisationcited by 7
  25. 2024NTK-Guided Few-Shot Class Incremental Learningcited by 7
  26. 2024The Effectiveness of Local Updates for Decentralized Learning under Data Heterogeneitycited by 7
  27. 2023Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networkscited by 6
  28. 2022Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widthscited by 5
  29. 2022Lazy Estimation of Variable Importance for Large Neural Networkscited by 5
  30. 2022Learning Lipschitz Functions by GD-trained Shallow Overparameterized ReLU Neural Networkscited by 5
  31. 2023Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widthscited by 5
  32. 2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
  33. 2024Theoretical characterisation of the Gauss-Newton conditioning in Neural Networkscited by 5
  34. 2025On the Convergence of Gradient Descent on Learning Transformers with Residual Connectionscited by 5
  35. 2026Most ReLU Networks Admit Identifiable Parameterscited by 5
  36. 2021How much pre-training is enough to discover a good subnetwork?cited by 4
  37. 2021Neural Network Gaussian Processes by Increasing Depthcited by 4
  38. 2022Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecturecited by 4

50 most cited of 87 All 87 (JSON)