# Learnable latent embeddings for joint behavioral and neural analysis

arXiv: https://arxiv.org/abs/2204.00673 · HTML: https://mlsanity.com/paper/2204.00673/

## Key facts

- Cited by 117 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 74 references that are in the map.
- First posted on arXiv 1 Apr 2022.

## Builds on (most cited first, 22 of 22)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning](https://mlsanity.com/paper/1805.08651/) (2018)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding](https://mlsanity.com/paper/2007.10930/) (2020)
- [On Linear Identifiability of Learned Representations](https://mlsanity.com/paper/2007.00810/) (2020)
- [Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA](https://mlsanity.com/paper/2106.09620/) (2021)
- [Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity](https://mlsanity.com/paper/2109.04463/) (2021)
- [Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE](https://mlsanity.com/paper/2011.04798/) (2020)
- [Large-scale neural recordings call for new insights to link brain and behavior](https://mlsanity.com/paper/2103.14662/) (2021)
- [Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity](https://mlsanity.com/paper/2107.04084/) (2021)
- [Demixed principal component analysis of population activity in higher cortical areas reveals independent representation of task parameters](https://mlsanity.com/paper/1410.6031/) (2014)
- [What can topology tell us about the neural code?](https://mlsanity.com/paper/1605.01905/) (2016)
- [Strong and weak principles of neural dimension reduction](https://mlsanity.com/paper/2011.08088/) (2020)
- [Decoding hand kinematics from population responses in sensorimotor cortex during grasping](https://mlsanity.com/paper/1904.03531/) (2019)

## Built on by (most cited first, 30 of 117)

- [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023)
- [MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data](https://mlsanity.com/paper/2403.11207/) (2024)
- [Decoding Natural Images from EEG for Object Recognition](https://mlsanity.com/paper/2308.13234/) (2023)
- [Rank-N-Contrast: Learning Continuous Representations for Regression](https://mlsanity.com/paper/2210.01189/) (2022)
- [Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity](https://mlsanity.com/paper/2305.11675/) (2023)
- [A Unified, Scalable Framework for Neural Population Decoding](https://mlsanity.com/paper/2310.16046/) (2023)
- [Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning](https://mlsanity.com/paper/2303.16535/) (2023)
- [Brain Captioning: Decoding human brain activity into images and text](https://mlsanity.com/paper/2305.11560/) (2023)
- [Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric, Topological, and Algebraic Structures](https://mlsanity.com/paper/2407.09468/) (2024)
- [Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder](https://mlsanity.com/paper/2402.17433/) (2024)
- [Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics](https://mlsanity.com/paper/2212.07892/) (2022)
- [NeuroBind: Towards Unified Multimodal Representations for Neural Signals](https://mlsanity.com/paper/2407.14020/) (2024)
- [A foundation model of vision, audition, and language for in-silico neuroscience](https://mlsanity.com/paper/2605.04326/) (2026)
- [Interpretable statistical representations of neural population dynamics and geometry](https://mlsanity.com/paper/2304.03376/) (2023)
- [AmadeusGPT: a natural language interface for interactive animal behavioral analysis](https://mlsanity.com/paper/2307.04858/) (2023)
- [Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data](https://mlsanity.com/paper/2311.00136/) (2023)
- [Subspace-Decomposed JEPAs: Disentangling Progression and Content in Latent World Models](https://mlsanity.com/paper/2605.31111/) (2026)
- [Decoding visual brain representations from electroencephalography through Knowledge Distillation and latent diffusion models](https://mlsanity.com/paper/2309.07149/) (2023)
- [Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data](https://mlsanity.com/paper/2310.03111/) (2023)
- [A Cross-Modal Approach to Silent Speech with LLM-Enhanced Recognition](https://mlsanity.com/paper/2403.05583/) (2024)
- [Words in Motion: Extracting Interpretable Control Vectors for Motion Transformers](https://mlsanity.com/paper/2406.11624/) (2024)
- [Exploring Behavior-Relevant and Disentangled Neural Dynamics with Generative Diffusion Models](https://mlsanity.com/paper/2410.09614/) (2024)
- [Self-supervised contrastive learning performs non-linear system identification](https://mlsanity.com/paper/2410.14673/) (2024)
- [Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research](https://mlsanity.com/paper/2504.13101/) (2025)
- [Generalizable, real-time neural decoding with hybrid state-space models](https://mlsanity.com/paper/2506.05320/) (2025)
- [Learning Time-Invariant Representations for Individual Neurons from Population Dynamics](https://mlsanity.com/paper/2311.02258/) (2023)
- [Meta-Dynamical State Space Models for Integrative Neural Data Analysis](https://mlsanity.com/paper/2410.05454/) (2024)
- [SPINT: Spatial Permutation-Invariant Neural Transformer for Consistent Intracortical Motor Decoding](https://mlsanity.com/paper/2507.08402/) (2025)
- [BRAID: Input-Driven Nonlinear Dynamical Modeling of Neural-Behavioral Data](https://mlsanity.com/paper/2509.18627/) (2025)
- [BaRISTA: Brain Scale Informed Spatiotemporal Representation of Human Intracranial Neural Activity](https://mlsanity.com/paper/2512.12135/) (2025)

## Abstract (arXiv)

Mapping behavioral actions to neural activity is a fundamental goal of neuroscience. As our ability to record large neural and behavioral data increases, there is growing interest in modeling neural dynamics during adaptive behaviors to probe neural representations. In particular, neural latent embeddings can reveal underlying correlates of behavior, yet, we lack non-linear techniques that can explicitly and flexibly leverage joint behavior and neural data. Here, we fill this gap with a novel method, CEBRA, that jointly uses behavioral and neural data in a hypothesis- or discovery-driven manner to produce consistent, high-performance latent spaces. We validate its accuracy and demonstrate our tool's utility for both calcium and electrophysiology datasets, across sensory and motor tasks, and in simple or complex behaviors across species. It allows for single and multi-session datasets to be leveraged for hypothesis testing or can be used label-free. Lastly, we show that CEBRA can be used for the mapping of space, uncovering complex kinematic features, and rapid, high-accuracy decoding of natural movies from visual cortex.