# Learning Scalable Deep Kernels with Recurrent Structure

arXiv: https://arxiv.org/abs/1610.08936 · HTML: https://mlsanity.com/paper/1610.08936/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2016.

## Built on by (most cited first, 22 of 22)

- [Dangers of Bayesian Model Averaging under Covariate Shift](https://mlsanity.com/paper/2106.11905/) (2021)
- [Diverse Video Generation using a Gaussian Process Trigger](https://mlsanity.com/paper/2107.04619/) (2021)
- [DKL-KAN: Scalable Deep Kernel Learning using Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2407.21176/) (2024)
- [Recent Advances in Data-Driven Wireless Communication Using Gaussian Processes: A Comprehensive Survey](https://mlsanity.com/paper/2103.10134/) (2021)
- [Energy-Based Models for Functional Data using Path Measure Tilting](https://mlsanity.com/paper/2202.01929/) (2022)
- [Calibrated simplex-mapping classification](https://mlsanity.com/paper/2103.02926/) (2021)
- [Stochastic Neural Networks with Infinite Width are Deterministic](https://mlsanity.com/paper/2201.12724/) (2022)
- [Gaussian Process Latent Variable Modeling for Few-shot Time Series Forecasting](https://mlsanity.com/paper/2212.10306/) (2022)
- [From Deep Additive Kernel Learning to Last-Layer Bayesian Neural Networks via Induced Prior Approximation](https://mlsanity.com/paper/2502.10540/) (2025)
- [Attentive Gaussian processes for probabilistic time-series generation](https://mlsanity.com/paper/2102.05208/) (2021)
- [Hierarchical Non-Stationary Temporal Gaussian Processes With L1-Regularization](https://mlsanity.com/paper/2105.09695/) (2021)
- [Deep Probabilistic Time Series Forecasting using Augmented Recurrent Input for Dynamic Systems](https://mlsanity.com/paper/2106.05848/) (2021)
- [Deep Learning Based Reconstruction of Total Solar Irradiance](https://mlsanity.com/paper/2107.11042/) (2021)
- [PCNN: A physics-constrained neural network for multiphase flows](https://mlsanity.com/paper/2109.08965/) (2021)
- [Fast Gaussian Process Posterior Mean Prediction via Local Cross Validation and Precomputation](https://mlsanity.com/paper/2205.10879/) (2022)
- [How Good are Low-Rank Approximations in Gaussian Process Regression?](https://mlsanity.com/paper/2112.06410/) (2021)
- [Deep Functional Factor Models: Forecasting High-Dimensional Functional Time Series via Bayesian Nonparametric Factorization](https://mlsanity.com/paper/2305.14543/) (2023)
- [Scalable Deep Basis Kernel Gaussian Processes](https://mlsanity.com/paper/2505.18526/) (2025)
- [A Class of Random-Kernel Network Models](https://mlsanity.com/paper/2509.01090/) (2025)
- [Response to Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2509.21228/) (2025)
- [Deep Kernel Learning for Stratifying Glaucoma Trajectories](https://mlsanity.com/paper/2605.00708/) (2026)
- [REGEN: Reference-Guided Synthetic Multivariate Time Series Generation for Forecasting](https://mlsanity.com/paper/2606.05264/) (2026)

## Abstract (arXiv)

Many applications in speech, robotics, finance, and biology deal with sequential data, where ordering matters and recurrent structures are common. However, this structure cannot be easily captured by standard kernel functions. To model such structure, we propose expressive closed-form kernel functions for Gaussian processes. The resulting model, GP-LSTM, fully encapsulates the inductive biases of long short-term memory (LSTM) recurrent networks, while retaining the non-parametric probabilistic advantages of Gaussian processes. We learn the properties of the proposed kernels by optimizing the Gaussian process marginal likelihood using a new provably convergent semi-stochastic gradient procedure and exploit the structure of these kernels for scalable training and prediction. This approach provides a practical representation for Bayesian LSTMs. We demonstrate state-of-the-art performance on several benchmarks, and thoroughly investigate a consequential autonomous driving application, where the predictive uncertainties provided by GP-LSTM are uniquely valuable.