# Recurrent Memory for Online Interdomain Gaussian Processes

arXiv: https://arxiv.org/abs/2502.08736 · HTML: https://mlsanity.com/paper/2502.08736/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [GP-VAE: Deep Probabilistic Time Series Imputation](https://mlsanity.com/paper/1907.04155/) (2019)
- [How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections](https://mlsanity.com/paper/2206.12037/) (2022)
- [Variational Fourier features for Gaussian processes](https://mlsanity.com/paper/1611.06740/) (2016)
- [Gaussian Process Prior Variational Autoencoders](https://mlsanity.com/paper/1810.11738/) (2018)
- [Rates of Convergence for Sparse Variational Gaussian Process Regression](https://mlsanity.com/paper/1903.03571/) (2019)
- [A Framework for Interdomain and Multioutput Gaussian Processes](https://mlsanity.com/paper/2003.01115/) (2020)
- [MCMC for Variationally Sparse Gaussian Processes](https://mlsanity.com/paper/1506.04000/) (2015)
- [Streaming Sparse Gaussian Process Approximations](https://mlsanity.com/paper/1705.07131/) (2017)
- [A Tutorial on Sparse Gaussian Processes and Variational Inference](https://mlsanity.com/paper/2012.13962/) (2020)
- [Variational Auto-Regressive Gaussian Processes for Continual Learning](https://mlsanity.com/paper/2006.05468/) (2020)
- [Conditioning Sparse Variational Gaussian Processes for Online Decision-making](https://mlsanity.com/paper/2110.15172/) (2021)
- [Sparse Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2010.10177/) (2020)
- [Scalable Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2010.13472/) (2020)
- [Kernel Interpolation for Scalable Online Gaussian Processes](https://mlsanity.com/paper/2103.01454/) (2021)
- [Spatial Mapping with Gaussian Processes and Nonstationary Fourier Features](https://mlsanity.com/paper/1711.05615/) (2017)
- [Markovian Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2207.05543/) (2022)
- [Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs](https://mlsanity.com/paper/1503.02424/) (2015)
- [Memory-Based Dual Gaussian Processes for Sequential Learning](https://mlsanity.com/paper/2306.03566/) (2023)
- [Sparse Algorithms for Markovian Gaussian Processes](https://mlsanity.com/paper/2103.10710/) (2021)

## Built on by (most cited first, 2 of 2)

- [Output Length Effect on DeepSeek-R1's Safety in Forced Thinking](https://mlsanity.com/paper/2503.01923/) (2025)
- [Interdomain Attention: Beyond Token-Level Key-Value Memory](https://mlsanity.com/paper/2605.24330/) (2026)

## Abstract (arXiv)

We propose a novel online Gaussian process (GP) model that is capable of capturing long-term memory in sequential data in an online learning setting. Our model, Online HiPPO Sparse Variational Gaussian Process (OHSVGP), leverages the HiPPO (High-order Polynomial Projection Operators) framework, which is popularized in the RNN domain due to its long-range memory modeling capabilities. We interpret the HiPPO time-varying orthogonal projections as inducing variables with time-dependent orthogonal polynomial basis functions, which allows the SVGP inducing variables to memorize the process history. We show that the HiPPO framework fits naturally into the interdomain GP framework and demonstrate that the kernel matrices can also be updated online in a recurrence form based on the ODE evolution of HiPPO. We evaluate OHSVGP with online prediction for 1D time series, continual learning in discriminative GP model for data with multidimensional inputs, and deep generative modeling with sparse Gaussian process variational autoencoder, showing that it outperforms existing online GP methods in terms of predictive performance, long-term memory preservation, and computational efficiency.