# Amortized Inference for Gaussian Process Hyperparameters of Structured Kernels

arXiv: https://arxiv.org/abs/2306.09819 · HTML: https://mlsanity.com/paper/2306.09819/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [GPflow: A Gaussian process library using TensorFlow](https://mlsanity.com/paper/1610.08733/) (2016)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [Structure Discovery in Nonparametric Regression through Compositional Kernel Search](https://mlsanity.com/paper/1302.4922/) (2013)
- [ChemBO: Bayesian Optimization of Small Organic Molecules with Synthesizable Recommendations](https://mlsanity.com/paper/1908.01425/) (2019)
- [Stochastic Hyperparameter Optimization through Hypernetworks](https://mlsanity.com/paper/1802.09419/) (2018)
- [Automatic Construction and Natural-Language Description of Nonparametric Regression Models](https://mlsanity.com/paper/1402.4304/) (2014)
- [Active Learning for Gaussian Process Considering Uncertainties with Application to Shape Control of Composite Fuselage](https://mlsanity.com/paper/2004.10931/) (2020)
- [Kernel Identification Through Transformers](https://mlsanity.com/paper/2106.08185/) (2021)
- [Safe Active Learning for Multi-Output Gaussian Processes](https://mlsanity.com/paper/2203.14849/) (2022)
- [Structural Kernel Search via Bayesian Optimization and Symbolical Optimal Transport](https://mlsanity.com/paper/2210.11836/) (2022)
- [Hierarchical-Hyperplane Kernels for Actively Learning Gaussian Process Models of Nonstationary Systems](https://mlsanity.com/paper/2303.10022/) (2023)

## Built on by (most cited first, 10 of 10)

- [Amortized In-Context Bayesian Posterior Estimation](https://mlsanity.com/paper/2502.06601/) (2025)
- [Amortized Safe Active Learning for Real-Time Data Acquisition: Pretrained Neural Policies From Simulated Nonparametric Functions](https://mlsanity.com/paper/2501.15458/) (2025)
- [Amortized nonmyopic active search via deep imitation learning](https://mlsanity.com/paper/2405.15031/) (2024)
- [Sparse Gaussian Neural Processes](https://mlsanity.com/paper/2504.01650/) (2025)
- [Future Aware Safe Active Learning of Time Varying Systems using Gaussian Processes](https://mlsanity.com/paper/2405.10581/) (2024)
- [Amortized Active Learning for Nonparametric Functions](https://mlsanity.com/paper/2407.17992/) (2024)
- [Batch Active Learning in Gaussian Process Regression using Derivatives](https://mlsanity.com/paper/2408.01861/) (2024)
- [Rapid AI-based generation of coverage paths for dispensing applications](https://mlsanity.com/paper/2505.03560/) (2025)
- [Modeling GRNs with a Probabilistic Categorical Framework](https://mlsanity.com/paper/2508.13208/) (2025)
- [Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks](https://mlsanity.com/paper/2601.21731/) (2026)

## Abstract (arXiv)

Learning the kernel parameters for Gaussian processes is often the computational bottleneck in applications such as online learning, Bayesian optimization, or active learning. Amortizing parameter inference over different datasets is a promising approach to dramatically speed up training time. However, existing methods restrict the amortized inference procedure to a fixed kernel structure. The amortization network must be redesigned manually and trained again in case a different kernel is employed, which leads to a large overhead in design time and training time. We propose amortizing kernel parameter inference over a complete kernel-structure-family rather than a fixed kernel structure. We do that via defining an amortization network over pairs of datasets and kernel structures. This enables fast kernel inference for each element in the kernel family without retraining the amortization network. As a by-product, our amortization network is able to do fast ensembling over kernel structures. In our experiments, we show drastically reduced inference time combined with competitive test performance for a large set of kernels and datasets.