# Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data

arXiv: https://arxiv.org/abs/2310.03111 · HTML: https://mlsanity.com/paper/2310.03111/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 18 of 18)

- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [GP-VAE: Deep Probabilistic Time Series Imputation](https://mlsanity.com/paper/1907.04155/) (2019)
- [Gaussian Process Kernels for Pattern Discovery and Extrapolation](https://mlsanity.com/paper/1302.4245/) (2013)
- [Learnable latent embeddings for joint behavioral and neural analysis](https://mlsanity.com/paper/2204.00673/) (2022)
- [Variational Fourier features for Gaussian processes](https://mlsanity.com/paper/1611.06740/) (2016)
- [Gaussian Process Prior Variational Autoencoders](https://mlsanity.com/paper/1810.11738/) (2018)
- [Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity](https://mlsanity.com/paper/2109.04463/) (2021)
- [Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE](https://mlsanity.com/paper/2011.04798/) (2020)
- [Longitudinal Variational Autoencoder](https://mlsanity.com/paper/2006.09763/) (2020)
- [Variational Latent Gaussian Process for Recovering Single-Trial Dynamics from Population Spike Trains](https://mlsanity.com/paper/1604.03053/) (2016)
- [Targeted Neural Dynamical Modeling](https://mlsanity.com/paper/2110.14853/) (2021)
- [Gacs-Korner Common Information Variational Autoencoder](https://mlsanity.com/paper/2205.12239/) (2022)
- [Linear-time inference for Gaussian Processes on one dimension](https://mlsanity.com/paper/2003.05554/) (2020)
- [Efficient non-conjugate Gaussian process factor models for spike count data using polynomial approximations](https://mlsanity.com/paper/1906.03318/) (2019)
- [A probabilistic framework for task-aligned intra- and inter-area neural manifold estimation](https://mlsanity.com/paper/2209.02816/) (2022)
- [Direct Estimation of Firing Rates from Calcium Imaging Data](https://mlsanity.com/paper/1601.00364/) (2016)

## Built on by (most cited first, 9 of 9)

- [Neural Encoding and Decoding at Scale](https://mlsanity.com/paper/2504.08201/) (2025)
- [Latent Diffusion for Neural Spiking Data](https://mlsanity.com/paper/2407.08751/) (2024)
- [Exploring Behavior-Relevant and Disentangled Neural Dynamics with Generative Diffusion Models](https://mlsanity.com/paper/2410.09614/) (2024)
- [BRAID: Input-Driven Nonlinear Dynamical Modeling of Neural-Behavioral Data](https://mlsanity.com/paper/2509.18627/) (2025)
- [Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data](https://mlsanity.com/paper/2509.18507/) (2025)
- [Cross-Modal Representational Knowledge Distillation for Enhanced Spike-Informed LFP Modeling](https://mlsanity.com/paper/2512.12461/) (2025)
- [Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference](https://mlsanity.com/paper/2512.12462/) (2025)
- [A Note on Generalization in Variational Autoencoders: How Effective Is Synthetic Data & Overparameterization?](https://mlsanity.com/paper/2310.19653/) (2023)
- [Unsupervised discovery of the shared and private geometry in multi-view data](https://mlsanity.com/paper/2408.12091/) (2024)

## Abstract (arXiv)

Characterizing the relationship between neural population activity and behavioral data is a central goal of neuroscience. While latent variable models (LVMs) are successful in describing high-dimensional time-series data, they are typically only designed for a single type of data, making it difficult to identify structure shared across different experimental data modalities. Here, we address this shortcoming by proposing an unsupervised LVM which extracts temporally evolving shared and independent latents for distinct, simultaneously recorded experimental modalities. We do this by combining Gaussian Process Factor Analysis (GPFA), an interpretable LVM for neural spiking data with temporally smooth latent space, with Gaussian Process Variational Autoencoders (GP-VAEs), which similarly use a GP prior to characterize correlations in a latent space, but admit rich expressivity due to a deep neural network mapping to observations. We achieve interpretability in our model by partitioning latent variability into components that are either shared between or independent to each modality. We parameterize the latents of our model in the Fourier domain, and show improved latent identification using this approach over standard GP-VAE methods. We validate our model on simulated multi-modal data consisting of Poisson spike counts and MNIST images that scale and rotate smoothly over time. We show that the multi-modal GP-VAE (MM-GPVAE) is able to not only identify the shared and independent latent structure across modalities accurately, but provides good reconstructions of both images and neural rates on held-out trials. Finally, we demonstrate our framework on two real world multi-modal experimental settings: Drosophila whole-brain calcium imaging alongside tracked limb positions, and Manduca sexta spike train measurements from ten wing muscles as the animal tracks a visual stimulus.