mlsanityData to 14 Oct 2026

Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels

arXiv 2306.03968 ↗First posted 6 Jun 2023stat.ML cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Selecting hyperparameters in deep learning greatly impacts its effectiveness but requires manual effort and expertise. Recent works show that Bayesian model selection with Laplace approximations can allow to optimize such hyperparameters just like standard neural network parameters using gradients and on the training data. However, estimating a single hyperparameter gradient requires a pass through the entire dataset, limiting the scalability of such algorithms. In this work, we overcome this issue by introducing lower bounds to the linearized Laplace approximation of the marginal likelihood. In contrast to previous estimators, these bounds are amenable to stochastic-gradient-based optimization and allow to trade off estimation accuracy against computational complexity. We derive them using the function-space form of the linearized Laplace, which can be estimated using the neural tangent kernel. Experimentally, we show that the estimators can significantly accelerate gradient-based hyperparameter optimization.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 37 of its 64 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

37 of 64 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Wide Residual Networkscited by 3,070
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2016Variational Inference: A Review for Statisticianscited by 1,472
  5. 2016Group Equivariant Convolutional Networkscited by 1,165
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2015Weight Uncertainty in Neural Networkscited by 554
  8. 2014New insights and perspectives on the natural gradient methodcited by 443
  9. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  10. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  11. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  12. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 169
25 more
  1. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  2. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 142
  3. 2018Three Mechanisms of Weight Decay Regularizationcited by 113
  4. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 106
  5. 2022Approximately Equivariant Networks for Imperfectly Symmetric Dynamicscited by 97
  6. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  7. 2015Distributed Gaussian Processescited by 84
  8. 2021The Promises and Pitfalls of Deep Kernel Learningcited by 81
  9. 2019BackPACK: Packing more into backpropcited by 78
  10. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  11. 2020Learning Invariances in Neural Networkscited by 52
  12. 2018Learning Invariances using the Marginal Likelihoodcited by 51
  13. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  14. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  15. 2018Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networkscited by 37
  16. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  17. 2022Accelerated Linearized Laplace Approximation for Bayesian Deep Learningcited by 28
  18. 2020Towards NNGP-guided Neural Architecture Searchcited by 24
  19. 2019Bayesian Image Classification with Deep Convolutional Gaussian Processescited by 21
  20. 2021Last Layer Marginal Likelihood for Invariance Learningcited by 21
  21. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
  22. 2022Learning Invariant Weights in Neural Networkscited by 16
  23. 2021Probing as Quantifying Inductive Biascited by 12
  24. 2023Hyperparameter Optimization through Neural Network Partitioningcited by 8
  25. 2022Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernelcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 3; 2026: 1.

10 papers.

  1. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  2. 2023Learning Layer-wise Equivariances Automatically using Gradientscited by 23
  3. 2024A Generative Model of Symmetry Transformationscited by 10
  4. 2024Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihoodcited by 7
  5. 2025Determinant Estimation under Memory Constraints and Neural Scaling Lawscited by 4
  6. 2023Online Laplace Model Selection Revisitedcited by 2
  7. 2024Gradients of Functions of Large Matricescited by 1
  8. 2025Spectral Estimation with Free Decompressioncited by 1
  9. 2025Flatness After All?cited by 1
  10. 2026Takeuchi's Information Criteria as Generalization Measures for DNNs Close to NTK Regimecited by 0