# Latent Diffusion for Neural Spiking Data

arXiv: https://arxiv.org/abs/2407.08751 · HTML: https://mlsanity.com/paper/2407.08751/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 27 of its 64 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation](https://mlsanity.com/paper/2107.03502/) (2021)
- [Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2101.12072/) (2021)
- [Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models](https://mlsanity.com/paper/2208.09399/) (2022)
- [Geometric Latent Diffusion Models for 3D Molecule Generation](https://mlsanity.com/paper/2305.01140/) (2023)
- [It's Raw! Audio Generation with State-Space Models](https://mlsanity.com/paper/2202.09729/) (2022)
- [Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding](https://mlsanity.com/paper/2211.06956/) (2022)
- [From Variational to Deterministic Autoencoders](https://mlsanity.com/paper/1903.12436/) (2019)
- [A Unified, Scalable Framework for Neural Population Decoding](https://mlsanity.com/paper/2310.16046/) (2023)
- [Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity](https://mlsanity.com/paper/2109.04463/) (2021)
- [Generalized Teacher Forcing for Learning Chaotic Dynamics](https://mlsanity.com/paper/2306.04406/) (2023)
- [Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE](https://mlsanity.com/paper/2011.04798/) (2020)
- [LFADS - Latent Factor Analysis via Dynamical Systems](https://mlsanity.com/paper/1608.06315/) (2016)
- [Variational Latent Gaussian Process for Recovering Single-Trial Dynamics from Population Spike Trains](https://mlsanity.com/paper/1604.03053/) (2016)
- [Enabling hyperparameter optimization in sequential autoencoders for spiking neural data](https://mlsanity.com/paper/1908.07896/) (2019)
- [Targeted Neural Dynamical Modeling](https://mlsanity.com/paper/2110.14853/) (2021)
- [Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data](https://mlsanity.com/paper/2311.00136/) (2023)
- [Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data](https://mlsanity.com/paper/2310.03111/) (2023)
- [lfads-torch: A modular and extensible implementation of latent factor analysis via dynamical systems](https://mlsanity.com/paper/2309.01230/) (2023)
- [Capturing the dynamical repertoire of single neurons with generalized linear models](https://mlsanity.com/paper/1602.07389/) (2016)
- [Temporal Conditioning Spiking Latent Variable Models of the Neural Response to Natural Visual Scenes](https://mlsanity.com/paper/2306.12045/) (2023)

## Built on by (most cited first, 12 of 12)

- [POCO: Scalable Neural Forecasting through Population Conditioning](https://mlsanity.com/paper/2506.14957/) (2025)
- [Negative Binomial Variational Autoencoders for Overdispersed Latent Modeling](https://mlsanity.com/paper/2508.05423/) (2025)
- [Contrastive Diffusion Alignment: Learning Structured Latents for Controllable Generation](https://mlsanity.com/paper/2510.14190/) (2025)
- [Energy-based Autoregressive Generation for Neural Population Dynamics](https://mlsanity.com/paper/2511.17606/) (2025)
- [Modeling Neural Activity with Conditionally Linear Dynamical Systems](https://mlsanity.com/paper/2502.18347/) (2025)
- [Cross-Session Decoding of Neural Spiking Data via Task-Conditioned Latent Alignment](https://mlsanity.com/paper/2601.19963/) (2026)
- [Modelling the Effects of Hearing Loss on Neural Coding in the Auditory Midbrain with Variational Conditioning](https://mlsanity.com/paper/2506.03088/) (2025)
- [DyC-STG: Dynamic Causal Spatio-Temporal Graph Network for Real-time Data Credibility Analysis in IoT](https://mlsanity.com/paper/2509.06483/) (2025)
- [Fast dynamical similarity analysis](https://mlsanity.com/paper/2511.22828/) (2025)
- [Machine Learning Methods for Studying Latent Neural Activity Dynamics](https://mlsanity.com/paper/2606.10530/) (2026)
- [A discrete generative model of neuronal spiking activity on microelectrode arrays](https://mlsanity.com/paper/2609.23907/) (2026)
- [Stable Neural Decoding Across Sessions via Task-Conditioned Latent Alignment for Brain-Machine Interfaces](https://mlsanity.com/paper/2609.27441/) (2026)

## Abstract (arXiv)

Modern datasets in neuroscience enable unprecedented inquiries into the relationship between complex behaviors and the activity of many simultaneously recorded neurons. While latent variable models can successfully extract low-dimensional embeddings from such recordings, using them to generate realistic spiking data, especially in a behavior-dependent manner, still poses a challenge. Here, we present Latent Diffusion for Neural Spiking data (LDNS), a diffusion-based generative model with a low-dimensional latent space: LDNS employs an autoencoder with structured state-space (S4) layers to project discrete high-dimensional spiking data into continuous time-aligned latents. On these inferred latents, we train expressive (conditional) diffusion models, enabling us to sample neural activity with realistic single-neuron and population spiking statistics. We validate LDNS on synthetic data, accurately recovering latent structure, firing rates, and spiking statistics. Next, we demonstrate its flexibility by generating variable-length data that mimics human cortical activity during attempted speech. We show how to equip LDNS with an expressive observation model that accounts for single-neuron dynamics not mediated by the latent state, further increasing the realism of generated samples. Finally, conditional LDNS trained on motor cortical activity during diverse reaching behaviors can generate realistic spiking data given reach direction or unseen reach trajectories. In summary, LDNS simultaneously enables inference of low-dimensional latents and realistic conditional generation of neural spiking datasets, opening up further possibilities for simulating experimentally testable hypotheses.