# Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors

arXiv: https://arxiv.org/abs/2410.16136 · HTML: https://mlsanity.com/paper/2410.16136/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 68 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 23 of 23)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Learnable latent embeddings for joint behavioral and neural analysis](https://mlsanity.com/paper/2204.00673/) (2022)
- [A Unified, Scalable Framework for Neural Population Decoding](https://mlsanity.com/paper/2310.16046/) (2023)
- [Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition](https://mlsanity.com/paper/1406.3284/) (2014)
- [Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE](https://mlsanity.com/paper/2011.04798/) (2020)
- [LFADS - Latent Factor Analysis via Dynamical Systems](https://mlsanity.com/paper/1608.06315/) (2016)
- [Neural system identification for large populations separating "what" and "where"](https://mlsanity.com/paper/1711.02653/) (2017)
- [A rotation-equivariant convolutional neural network model of primary visual cortex](https://mlsanity.com/paper/1809.10504/) (2018)
- [The Sensorium competition on predicting large-scale mouse primary visual cortex activity](https://mlsanity.com/paper/2206.08666/) (2022)
- [The implications of perception as probabilistic inference for correlated neural variability during behavior](https://mlsanity.com/paper/1409.0257/) (2014)
- [The Dynamic Sensorium competition for predicting large-scale mouse visual cortex activity from videos](https://mlsanity.com/paper/2305.19654/) (2023)
- [Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data](https://mlsanity.com/paper/2311.00136/) (2023)
- [V1T: large-scale mouse V1 response prediction using a Vision Transformer](https://mlsanity.com/paper/2302.03023/) (2023)
- [Retrospective for the Dynamic Sensorium Competition for predicting large-scale mouse primary visual cortex activity from videos](https://mlsanity.com/paper/2407.09100/) (2024)
- [Explaining V1 Properties with a Biologically Constrained Deep Learning Architecture](https://mlsanity.com/paper/2305.11275/) (2023)
- [Understanding Neural Coding on Latent Manifolds by Sharing Features and Dividing Ensembles](https://mlsanity.com/paper/2210.03155/) (2022)
- [A zero-inflated gamma model for deconvolved calcium imaging traces](https://mlsanity.com/paper/2006.03737/) (2020)
- [Reproducibility of predictive networks for mouse visual cortex](https://mlsanity.com/paper/2406.12625/) (2024)
- [Learning low-dimensional dynamics from whole-brain data improves task capture](https://mlsanity.com/paper/2305.14369/) (2023)
- [Probabilistic Neural Transfer Function Estimation with Bayesian System Identification](https://mlsanity.com/paper/2308.05990/) (2023)
- [Predictive variational autoencoder for learning robust representations of time-series data](https://mlsanity.com/paper/2312.06932/) (2023)

## Built on by (most cited first, 1 of 1)

- [OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens](https://mlsanity.com/paper/2604.18827/) (2026)

## Abstract (arXiv)

The neural activity in the visual processing is influenced by both external stimuli and internal brain states. Ideally, a neural predictive model should account for both of them. Currently, there are no dynamic encoding models that explicitly model a latent state and the entire neuronal response distribution. We address this gap by proposing a probabilistic model that predicts the joint distribution of the neuronal responses from video stimuli and stimulus-independent latent factors. After training and testing our model on mouse V1 neuronal responses, we find that it outperforms video-only models in terms of log-likelihood and achieves improvements in likelihood and correlation when conditioned on responses from other neurons. Furthermore, we find that the learned latent factors strongly correlate with mouse behavior and that they exhibit patterns related to the neurons' position on the visual cortex, although the model was trained without behavior and cortical coordinates. Our findings demonstrate that unsupervised learning of latent factors from population responses can reveal biologically meaningful structure that bridges sensory processing and behavior, without requiring explicit behavioral annotations during training.