# Neighbour-Driven Gaussian Process Variational Autoencoders for Scalable Structured Latent Modelling

arXiv: https://arxiv.org/abs/2505.16481 · HTML: https://mlsanity.com/paper/2505.16481/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 27 of 27)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [GP-VAE: Deep Probabilistic Time Series Imputation](https://mlsanity.com/paper/1907.04155/) (2019)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [Handling Incomplete Heterogeneous Data using VAEs](https://mlsanity.com/paper/1807.03653/) (2018)
- [Hierarchical Nearest-Neighbor Gaussian Process Models for Large Geostatistical Datasets](https://mlsanity.com/paper/1406.7343/) (2014)
- [Gaussian Process Prior Variational Autoencoders](https://mlsanity.com/paper/1810.11738/) (2018)
- [A general framework for Vecchia approximations of Gaussian processes](https://mlsanity.com/paper/1708.06302/) (2017)
- [MCMC for Variationally Sparse Gaussian Processes](https://mlsanity.com/paper/1506.04000/) (2015)
- [Spatio-Temporal Variational Gaussian Processes](https://mlsanity.com/paper/2111.01732/) (2021)
- [Sparse Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2010.10177/) (2020)
- [Fast Variational Learning in State-Space Gaussian Process Models](https://mlsanity.com/paper/2007.04731/) (2020)
- [Scalable Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2010.13472/) (2020)
- [Sparse within Sparse Gaussian Processes using Neighbor Information](https://mlsanity.com/paper/2011.05041/) (2020)
- [Markovian Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2207.05543/) (2022)
- [Doubly Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2001.05363/) (2020)
- [Correlation-based sparse inverse Cholesky factorization for fast Gaussian-process inference](https://mlsanity.com/paper/2112.14591/) (2021)
- [Model Selection for Bayesian Autoencoders](https://mlsanity.com/paper/2106.06245/) (2021)
- [Fully Bayesian Autoencoders with Latent Sparse Gaussian Processes](https://mlsanity.com/paper/2302.04534/) (2023)
- [Leveraging Locality and Robustness to Achieve Massively Scalable Gaussian Process Regression](https://mlsanity.com/paper/2306.14731/) (2023)
- [Variational sparse inverse Cholesky approximation for latent Gaussian processes via double Kullback-Leibler minimization](https://mlsanity.com/paper/2301.13303/) (2023)

## Built on by (most cited first, 3 of 3)

- [Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?](https://mlsanity.com/paper/2602.06500/) (2026)
- [Modeling Temporal scRNA-seq Data with Latent Gaussian Process and Optimal Transport](https://mlsanity.com/paper/2605.20989/) (2026)
- [Gaussian Process Prior Variational Autoencoder for Endoscopic Videos](https://mlsanity.com/paper/2606.19908/) (2026)

## Abstract (arXiv)

Gaussian Process (GP) Variational Autoencoders (VAEs) extend standard VAEs by replacing the fully factorised Gaussian prior with a GP prior, thereby capturing richer correlations among latent variables. However, performing exact GP inference in large-scale GPVAEs is computationally prohibitive, often forcing existing approaches to rely on restrictive kernel assumptions or large sets of inducing points. In this work, we propose a neighbour-driven approximation strategy that exploits local adjacencies in the latent space to achieve scalable GPVAE inference. By confining computations to the nearest neighbours of each data point, our method preserves essential latent dependencies, allowing more flexible kernel choices and mitigating the need for numerous inducing points. Through extensive experiments on tasks including representation learning, data imputation, and conditional generation, we demonstrate that our approach outperforms other GPVAE variants in both predictive performance and computational efficiency.