# Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity

arXiv: https://arxiv.org/abs/2109.04463 · HTML: https://mlsanity.com/paper/2109.04463/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 99 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Representation learning for neural population activity with Neural Data Transformers](https://mlsanity.com/paper/2108.01210/) (2021)
- [Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE](https://mlsanity.com/paper/2011.04798/) (2020)
- [Machine learning for neural decoding](https://mlsanity.com/paper/1708.00909/) (2017)
- [Hierarchical Optimal Transport for Multimodal Distribution Alignment](https://mlsanity.com/paper/1906.11768/) (2019)
- [Linear dynamical neural population models through nonlinear embeddings](https://mlsanity.com/paper/1605.08454/) (2016)
- [EvalAI: Towards Better Evaluation Systems for AI Agents](https://mlsanity.com/paper/1902.03570/) (2019)
- [Adversarial Domain Adaptation for Stable Brain-Machine Interfaces](https://mlsanity.com/paper/1810.00045/) (2018)
- [Tree-Structured Recurrent Switching Linear Dynamical Systems for Multi-Scale Modeling](https://mlsanity.com/paper/1811.12386/) (2018)
- [Variational Latent Gaussian Process for Recovering Single-Trial Dynamics from Population Spike Trains](https://mlsanity.com/paper/1604.03053/) (2016)
- [Enabling hyperparameter optimization in sequential autoencoders for spiking neural data](https://mlsanity.com/paper/1908.07896/) (2019)
- [On Simplicity and Complexity in the Brave New World of Large-Scale Neuroscience](https://mlsanity.com/paper/1503.08779/) (2015)
- [Manifold GPLVMs for discovering non-Euclidean latent structure in neural data](https://mlsanity.com/paper/2006.07429/) (2020)
- [Neural Dynamics Discovery via Gaussian Process Recurrent Neural Networks](https://mlsanity.com/paper/1907.00650/) (2019)
- [Poisson-Randomized Gamma Dynamical Systems](https://mlsanity.com/paper/1910.12991/) (2019)
- [Building population models for large-scale neural recordings: opportunities and pitfalls](https://mlsanity.com/paper/2102.01807/) (2021)
- [Deep Random Splines for Point Process Intensity Estimation of Neural Population Data](https://mlsanity.com/paper/1903.02610/) (2019)
- [Efficient non-conjugate Gaussian process factor models for spike count data using polynomial approximations](https://mlsanity.com/paper/1906.03318/) (2019)
- [Extracting low-dimensional dynamics from multiple large-scale neural population recordings by learning to predict correlations](https://mlsanity.com/paper/1711.01847/) (2017)
- [Incorporating structured assumptions with probabilistic graphical models in fMRI data analysis](https://mlsanity.com/paper/2005.04879/) (2020)

## Built on by (most cited first, 30 of 60)

- [Learnable latent embeddings for joint behavioral and neural analysis](https://mlsanity.com/paper/2204.00673/) (2022)
- [A Unified, Scalable Framework for Neural Population Decoding](https://mlsanity.com/paper/2310.16046/) (2023)
- [Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution](https://mlsanity.com/paper/2407.14668/) (2024)
- [STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer](https://mlsanity.com/paper/2206.04727/) (2022)
- [Neural Encoding and Decoding at Scale](https://mlsanity.com/paper/2504.08201/) (2025)
- [Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers](https://mlsanity.com/paper/2206.06131/) (2022)
- [Inferring stochastic low-rank recurrent neural networks from neural data](https://mlsanity.com/paper/2406.16749/) (2024)
- [The Sensorium competition on predicting large-scale mouse primary visual cortex activity](https://mlsanity.com/paper/2206.08666/) (2022)
- [Latent Diffusion for Neural Spiking Data](https://mlsanity.com/paper/2407.08751/) (2024)
- [ZAPBench: A Benchmark for Whole-Brain Activity Prediction in Zebrafish](https://mlsanity.com/paper/2503.02618/) (2025)
- [Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data](https://mlsanity.com/paper/2310.03111/) (2023)
- [Generalizable, real-time neural decoding with hybrid state-space models](https://mlsanity.com/paper/2506.05320/) (2025)
- [lfads-torch: A modular and extensible implementation of latent factor analysis via dynamical systems](https://mlsanity.com/paper/2309.01230/) (2023)
- [Learning Time-Invariant Representations for Individual Neurons from Population Dynamics](https://mlsanity.com/paper/2311.02258/) (2023)
- [Brain-to-Text Benchmark '24: Lessons Learned](https://mlsanity.com/paper/2412.17227/) (2024)
- [eXponential FAmily Dynamical Systems (XFADS): Large-scale nonlinear Gaussian state-space modeling](https://mlsanity.com/paper/2403.01371/) (2024)
- [POCO: Scalable Neural Forecasting through Population Conditioning](https://mlsanity.com/paper/2506.14957/) (2025)
- [BaRISTA: Brain Scale Informed Spatiotemporal Representation of Human Intracranial Neural Activity](https://mlsanity.com/paper/2512.12135/) (2025)
- [Expressive architectures enhance interpretability of dynamics-based neural population models](https://mlsanity.com/paper/2212.03771/) (2022)
- [Recurrences reveal shared causal drivers of complex time series](https://mlsanity.com/paper/2301.13516/) (2023)
- [Linear Time GPs for Inferring Latent Trajectories from Neural Spike Trains](https://mlsanity.com/paper/2306.01802/) (2023)
- [SiBBlInGS: Similarity-driven Building-Block Inference using Graphs across States](https://mlsanity.com/paper/2306.04817/) (2023)
- [Neural Latent Aligner: Cross-trial Alignment for Learning Representations of Complex, Naturalistic Neural Data](https://mlsanity.com/paper/2308.06443/) (2023)
- [Multi-Region Markovian Gaussian Process: An Efficient Method to Discover Directional Communications Across Multiple Brain Regions](https://mlsanity.com/paper/2402.02686/) (2024)
- [Animal behavioral analysis and neural encoding with transformer-based self-supervised pretraining](https://mlsanity.com/paper/2507.09513/) (2025)
- [BLEND: Behavior-guided Neural Population Dynamics Modeling via Privileged Knowledge Distillation](https://mlsanity.com/paper/2410.13872/) (2024)
- [Generative Modeling of Neural Dynamics via Latent Stochastic Differential Equations](https://mlsanity.com/paper/2412.12112/) (2024)
- [Understanding Neural Coding on Latent Manifolds by Sharing Features and Dividing Ensembles](https://mlsanity.com/paper/2210.03155/) (2022)
- [Real-Time Variational Method for Learning Neural Trajectory and its Dynamics](https://mlsanity.com/paper/2305.11278/) (2023)
- [Graph Laplacian Learning with Exponential Family Noise](https://mlsanity.com/paper/2306.08201/) (2023)

## Abstract (arXiv)

Advances in neural recording present increasing opportunities to study neural activity in unprecedented detail. Latent variable models (LVMs) are promising tools for analyzing this rich activity across diverse neural systems and behaviors, as LVMs do not depend on known relationships between the activity and external experimental variables. However, progress with LVMs for neuronal population activity is currently impeded by a lack of standardization, resulting in methods being developed and compared in an ad hoc manner. To coordinate these modeling efforts, we introduce a benchmark suite for latent variable modeling of neural population activity. We curate four datasets of neural spiking activity from cognitive, sensory, and motor areas to promote models that apply to the wide variety of activity seen across these areas. We identify unsupervised evaluation as a common framework for evaluating models across datasets, and apply several baselines that demonstrate benchmark diversity. We release this benchmark through EvalAI. http://neurallatents.github.io