mlsanityData to 7 Oct 2026

Kernel Identification Through Transformers

arXiv 2106.08185 ↗First posted 15 Jun 2021stat.ML cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Kernel selection plays a central role in determining the performance of Gaussian Process (GP) models, as the chosen kernel determines both the inductive biases and prior support of functions under the GP prior. This work addresses the challenge of constructing custom kernel functions for high-dimensional GP regression models. Drawing inspiration from recent progress in deep learning, we introduce a novel approach named KITT: Kernel Identification Through Transformers. KITT exploits a transformer-based architecture to generate kernel recommendations in under 0.1 seconds, which is several orders of magnitude faster than conventional kernel search algorithms. We train our model using synthetic data generated from priors over a vocabulary of known kernels. By exploiting the nature of the self-attention mechanism, KITT is able to process datasets with inputs of arbitrary dimension. We demonstrate that kernels chosen by KITT yield strong performance over a diverse collection of regression benchmarks.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

20 of 36 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2016Layer Normalizationcited by 5,579
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  8. 2017Deep Setscited by 1,394
  9. 2015Deep Kernel Learningcited by 440
  10. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  11. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  12. 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 129
8 more
  1. 2013Structure Discovery in Nonparametric Regression through Compositional Kernel Searchcited by 86
  2. 2019Multimodal Transformer with Multi-View Visual Representation for Image Captioningcited by 86
  3. 2021The Promises and Pitfalls of Deep Kernel Learningcited by 81
  4. 2014Manifold Gaussian Processes for Regressioncited by 79
  5. 2021Efficient Transformers in Reinforcement Learning using Actor-Learner Distillationcited by 35
  6. 2018Differentiable Compositional Kernel Learning for Gaussian Processescited by 23
  7. 2017Scaling up the Automatic Statistician: Scalable Structure Discovery using Gaussian Processescited by 19
  8. 2021The Minecraft Kernel: Modelling correlated Gaussian Processes in the Fourier domaincited by 3

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 5; 2024: 2; 2025: 4.

13 papers.

Sort:

  1. 2022Neural Diffusion Processescited by 50
  2. 2024Amortized Probabilistic Conditioning for Optimization, Simulation and Inferencecited by 22
  3. 2025Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMscited by 15
  4. 2023Practical Equivariances via Relational Conditional Neural Processescited by 13
  5. 2025Amortized In-Context Bayesian Posterior Estimationcited by 13
  6. 2023Amortized Inference for Gaussian Process Hyperparameters of Structured Kernelscited by 10
  7. 2024EARL-BO: Reinforcement Learning for Multi-Step Lookahead, High-Dimensional Bayesian Optimizationcited by 6
  8. 2022Gaussian Process Surrogate Models for Neural Networkscited by 5
  9. 2023MONGOOSE: Path-wise Smooth Bayesian Optimisation via Meta-learningcited by 5
  10. 2023On the Identifiability and Interpretability of Gaussian Process Modelscited by 2
  11. 2023Beyond Intuition, a Framework for Applying GPs to Real-World Datacited by 1
  12. 2025Meta-Statistical Learning: Supervised Learning of Statistical Estimatorscited by 1
1 more
  1. 2025MIST: Mutual Information Estimation Via Supervised Trainingcited by 1