# Consistency of Empirical Bayes And Kernel Flow For Hierarchical Parameter Estimation

arXiv: https://arxiv.org/abs/2005.11375 · HTML: https://mlsanity.com/paper/2005.11375/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2020.

## Built on by (most cited first, 18 of 18)

- [Solving and Learning Nonlinear PDEs with Gaussian Processes](https://mlsanity.com/paper/2103.12959/) (2021)
- [Kernel Methods are Competitive for Operator Learning](https://mlsanity.com/paper/2304.13202/) (2023)
- [Sparse Cholesky Factorization for Solving Nonlinear PDEs via Gaussian Processes](https://mlsanity.com/paper/2304.01294/) (2023)
- [Computational Graph Completion](https://mlsanity.com/paper/2110.10323/) (2021)
- [Error Analysis of Kernel/GP Methods for Nonlinear and Parametric PDEs](https://mlsanity.com/paper/2305.04962/) (2023)
- [Sampling via Gradient Flows in the Space of Probability Measures](https://mlsanity.com/paper/2310.03597/) (2023)
- [Maximum Likelihood Estimation in Gaussian Process Regression is Ill-Posed](https://mlsanity.com/paper/2203.09179/) (2022)
- [One-Shot Learning of Stochastic Differential Equations with Data Adapted Kernels](https://mlsanity.com/paper/2209.12086/) (2022)
- [Learning dynamical systems from data: A simple cross-validation perspective, part III: Irregularly-Sampled Time Series](https://mlsanity.com/paper/2111.13037/) (2021)
- [Nonparametric learning of kernels in nonlocal operators](https://mlsanity.com/paper/2205.11006/) (2022)
- [Learning "best" kernels from data in Gaussian process regression. With application to aerodynamics](https://mlsanity.com/paper/2206.02563/) (2022)
- [Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features](https://mlsanity.com/paper/2407.00584/) (2024)
- [Gaussian Process Hydrodynamics](https://mlsanity.com/paper/2209.10707/) (2022)
- [Gaussian Measures Conditioned on Nonlinear Observations: Consistency, MAP Estimators, and Simulation](https://mlsanity.com/paper/2405.13149/) (2024)
- [Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processes](https://mlsanity.com/paper/2510.05568/) (2025)
- [Asymptotic Bounds for Smoothness Parameter Estimates in Gaussian Process Interpolation](https://mlsanity.com/paper/2203.05400/) (2022)
- [Solving Functional PDEs with Gaussian Processes and Applications to Functional Renormalization Group Equations](https://mlsanity.com/paper/2512.20956/) (2025)
- [On the Nonasymptotic Scaling Guarantee of Hyperparameter Estimation in Inhomogeneous, Weakly-Dependent Complex Network Dynamical Systems](https://mlsanity.com/paper/2601.15603/) (2026)

## Abstract (arXiv)

Gaussian process regression has proven very powerful in statistics, machine learning and inverse problems. A crucial aspect of the success of this methodology, in a wide range of applications to complex and real-world problems, is hierarchical modeling and learning of hyperparameters. The purpose of this paper is to study two paradigms of learning hierarchical parameters: one is from the probabilistic Bayesian perspective, in particular, the empirical Bayes approach that has been largely used in Bayesian statistics; the other is from the deterministic and approximation theoretic view, and in particular the kernel flow algorithm that was proposed recently in the machine learning literature. Analysis of their consistency in the large data limit, as well as explicit identification of their implicit bias in parameter learning, are established in this paper for a Mat'ern-like model on the torus. A particular technical challenge we overcome is the learning of the regularity parameter in the Mat'ern-like field, for which consistency results have been very scarce in the spatial statistics literature. Moreover, we conduct extensive numerical experiments beyond the Mat'ern-like model, comparing the two algorithms further. These experiments demonstrate learning of other hierarchical parameters, such as amplitude and lengthscale; they also illustrate the setting of model misspecification in which the kernel flow approach could show superior performance to the more traditional empirical Bayes approach.