A note on regularised NTK dynamics with an application to PAC-Bayesian training
Not on the map. Why?
Abstract (arXiv)
We establish explicit dynamics for neural networks whose training objective has a regularising term that constrains the parameters to remain close to their initial value. This keeps the network in a lazy training regime, where the dynamics can be linearised around the initialisation. The standard neural tangent kernel (NTK) governs the evolution during the training in the infinite-width limit, although the regularisation yields an additional term appears in the differential equation describing the dynamics. This setting provides an appropriate framework to study the evolution of wide networks trained to optimise generalisation objectives such as PAC-Bayes bounds, and hence potentially contribute to a deeper theoretical understanding of such networks.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 48 of its 61 references that are in the map. First posted on arXiv 20 Dec 2023.
Builds on
48 of 61 references.
References · 48
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2017Deep Neural Networks as Gaussian Processescited by 622
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
- 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
- 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
36 more
- 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
- 2021User-friendly introduction to PAC-Bayes boundscited by 239
- 2020Feature Learning in Infinite-Width Neural Networkscited by 190
- 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
- 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
- 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
- 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
- 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
- 2019A Primer on PAC-Bayesian Learningcited by 132
- 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
- 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 126
- 2015On the properties of variational approximations of Gibbs posteriorscited by 121
- 2020Tighter risk certificates for neural networkscited by 112
- 2017Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priorscited by 81
- 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 76
- 2020On the role of data in PAC-Bayes boundscited by 64
- 2023Generalization Bounds: Perspectives from Information Theory and PAC-Bayescited by 61
- 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
- 2019Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resiliencecited by 51
- 2020Stable ResNetcited by 49
- 2019Dichotomize and Generalize: PAC-Bayesian Binary Activated Deep Neural Networkscited by 38
- 2020Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networkscited by 29
- 2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 25
- 2019Finite size corrections for neural network Gaussian processescited by 22
- 2022Non-Vacuous Generalisation Bounds for Shallow Neural Networkscited by 22
- 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
- 2021Learning PAC-Bayes Priors for Probabilistic Neural Networkscited by 21
- 2019PAC-Bayesian Contrastive Unsupervised Representation Learningcited by 20
- 2022On PAC-Bayesian reconstruction guarantees for VAEscited by 20
- 2019Convex Formulation of Overparameterized Deep Neural Networkscited by 16
- 2023Learning via Wasserstein-Based High Probability Generalisation Boundscited by 12
- 2021Wide stochastic networks: Gaussian limit and PAC-Bayesian trainingcited by 10
- 2021Progress in Self-Certified Neural Networkscited by 10
- 2022Generalisation under gradient descent via deterministic PAC-Bayescited by 9
- 2021Conditionally Gaussian PAC-Bayescited by 7
- 2022Demystify Optimization and Generalization of Over-parameterized PAC-Bayesian Learningcited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.