mlsanityData to 7 Oct 2026

Second-Order Path Kernel Interpolation Formulas in Machine Learning

arXiv 2606.07495 ↗First posted 5 Jun 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Understanding how training data shape neural network predictions is a central problem in modern learning theory. In 2020, Pedro Domingos proposed an interpolation formula valid for every model learned by deterministic gradient descent. It expresses the model's prediction as an integral, along the optimization path, of a data-dependent kernel that aligns the model's gradients at the test and training data. Such a first-order characterization remains valid for models trained with batch-based stochastic optimization. In this paper, we develop second-order forms of these interpolation formulas. We show that the leading path-kernel interpolation is supplemented by a curvature-weighted interpolation term. For stochastic gradient descent, an additional sampling-induced component appears, coupling the curvature of the prediction with the covariance of mini-batch gradient noise. We also extend the representation to stochastic gradient descent with momentum, where the interpolation structure is preserved but with the weights modified by a memory-related factor. Moreover, we establish a concentration estimate for the terminal prediction, identifying the fluctuation scale around the expected second-order representation. Together, these results provide a refinement of the path-kernel interpretation of neural network prediction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

25 of 38 references.

References · 25

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  4. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  7. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  8. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  9. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  10. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  11. 2017Three Factors Influencing Minima in SGDcited by 270
  12. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
13 more
  1. 2020Implicit Gradient Regularizationcited by 145
  2. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  3. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  4. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  5. 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 56
  6. 2020Every Model Learned by Gradient Descent Is Approximately a Kernel Machinecited by 53
  7. 2023Why (and When) does Local SGD Generalize Better than SGD?cited by 26
  8. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
  9. 2019Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximationcited by 14
  10. 2023Backward error analysis and the qualitative behaviour of stochastic optimization algorithms: Application to stochastic coordinate descentcited by 5
  11. 2024Curvature Clues: Decoding Deep Learning Privacy with Input Loss Curvaturecited by 4
  12. 2024A Hessian-Aware Stochastic Differential Equation for Modelling SGDcited by 2
  13. 2026On Interpolation Formulas Describing Neural Network Generalizationcited by 1

Built on by

Not cited yet.