# Exploring Behavior-Relevant and Disentangled Neural Dynamics with Generative Diffusion Models

arXiv: https://arxiv.org/abs/2410.09614 · HTML: https://mlsanity.com/paper/2410.09614/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 37 of its 64 references that are in the map.
- First posted on arXiv 12 Oct 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 37)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Flexible Diffusion Modeling of Long Videos](https://mlsanity.com/paper/2205.11495/) (2022)
- [A Variational Perspective on Solving Inverse Problems with Diffusion Models](https://mlsanity.com/paper/2305.04391/) (2023)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Learnable latent embeddings for joint behavioral and neural analysis](https://mlsanity.com/paper/2204.00673/) (2022)
- [Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks](https://mlsanity.com/paper/2311.10944/) (2023)
- [Variational Recurrent Auto-Encoders](https://mlsanity.com/paper/1412.6581/) (2014)
- [Learning Latent Subspaces in Variational Autoencoders](https://mlsanity.com/paper/1812.06190/) (2018)
- [Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE](https://mlsanity.com/paper/2011.04798/) (2020)
- [Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities](https://mlsanity.com/paper/2305.17214/) (2023)
- [Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance](https://mlsanity.com/paper/2208.08664/) (2022)
- [Targeted Neural Dynamical Modeling](https://mlsanity.com/paper/2110.14853/) (2021)
- [Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity](https://mlsanity.com/paper/2111.02338/) (2021)
- [Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers](https://mlsanity.com/paper/2206.06131/) (2022)
- [Measuring behavior across scales](https://mlsanity.com/paper/1712.05784/) (2017)
- [NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties](https://mlsanity.com/paper/2402.01590/) (2024)
- [Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data](https://mlsanity.com/paper/2310.03111/) (2023)
- [Extraction and Recovery of Spatio-Temporal Structure in Latent Dynamics Alignment with Diffusion Models](https://mlsanity.com/paper/2306.06138/) (2023)

## Built on by (most cited first, 9 of 9)

- [How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook](https://mlsanity.com/paper/2503.11835/) (2025)
- [Your contrastive learning problem is secretly a distribution alignment problem](https://mlsanity.com/paper/2502.20141/) (2025)
- [Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data](https://mlsanity.com/paper/2509.18507/) (2025)
- [A Revisit of Total Correlation in Disentangled Variational Auto-Encoder with Partial Disentanglement](https://mlsanity.com/paper/2502.02279/) (2025)
- [BrainStratify: Coarse-to-Fine Disentanglement of Intracranial Neural Dynamics](https://mlsanity.com/paper/2505.20480/) (2025)
- [Cross-Subject Modeling for Widefield Calcium Imaging via Atlas-Aligned Spatiotemporal Tokenization](https://mlsanity.com/paper/2607.09754/) (2026)
- [Uncovering Semantic Selectivity of Latent Groups in Higher Visual Cortex with Mutual Information-Guided Diffusion](https://mlsanity.com/paper/2510.02182/) (2025)
- [ALIGN: Adversarial Learning for Generalizable Speech Neuroprosthesis](https://mlsanity.com/paper/2603.18299/) (2026)
- [Machine Learning Methods for Studying Latent Neural Activity Dynamics](https://mlsanity.com/paper/2606.10530/) (2026)

## Abstract (arXiv)

Understanding the neural basis of behavior is a fundamental goal in neuroscience. Current research in large-scale neuro-behavioral data analysis often relies on decoding models, which quantify behavioral information in neural data but lack details on behavior encoding. This raises an intriguing scientific question: ``how can we enable in-depth exploration of neural representations in behavioral tasks, revealing interpretable neural dynamics associated with behaviors''. However, addressing this issue is challenging due to the varied behavioral encoding across different brain regions and mixed selectivity at the population level. To tackle this limitation, our approach, named ``BeNeDiff'', first identifies a fine-grained and disentangled neural subspace using a behavior-informed latent variable model. It then employs state-of-the-art generative diffusion models to synthesize behavior videos that interpret the neural dynamics of each latent factor. We validate the method on multi-session datasets containing widefield calcium imaging recordings across the dorsal cortex. Through guiding the diffusion model to activate individual latent factors, we verify that the neural dynamics of latent factors in the disentangled neural subspace provide interpretable quantifications of the behaviors of interest. At the same time, the neural subspace in BeNeDiff demonstrates high disentanglement and neural reconstruction quality.