mlsanityData to 14 Oct 2026

Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference

arXiv 2512.12462 ↗First posted 13 Dec 2025cs.LG cs.AI q-bio.NCcited by 2

Not on the map. Why?

Abstract (arXiv)

Real-time decoding of target variables from multiple simultaneously recorded neural time-series modalities, such as discrete spiking activity and continuous field potentials, is important across various neuroscience applications. However, a major challenge for doing so is that different neural modalities can have different timescales (i.e., sampling rates) and different probabilistic distributions, or can even be missing at some time-steps. Existing nonlinear models of multimodal neural activity do not address different timescales or missing samples across modalities. Further, some of these models do not allow for real-time decoding. Here, we develop a learning framework that can enable real-time recursive decoding while nonlinearly aggregating information across multiple modalities with different timescales and distributions and with missing samples. This framework consists of 1) a multiscale encoder that nonlinearly aggregates information after learning within-modality dynamics to handle different timescales and missing samples in real time, 2) a multiscale dynamical backbone that extracts multimodal temporal dynamics and enables real-time recursive decoding, and 3) modality-specific decoders to account for different probabilistic distributions across modalities. In both simulations and three distinct multiscale brain datasets, we show that our model can aggregate information across modalities with different timescales and distributions and missing samples to improve real-time target decoding. Further, our method outperforms various linear and nonlinear multimodal benchmarks in doing so.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 32 of its 95 references that are in the map. First posted on arXiv 13 Dec 2025.

Builds on

32 of 95 references.

References · 32

  1. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  2. 2015Generating Sentences from a Continuous Spacecited by 612
  3. 2016Recurrent Neural Networks for Multivariate Time Series with Missing Valuescited by 554
  4. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 222
  5. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  6. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 178
  7. 2018Learning Factorized Multimodal Representationscited by 171
  8. 2022Learnable latent embeddings for joint behavioral and neural analysiscited by 117
  9. 2017A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learningcited by 107
  10. 2016Joint Multimodal Learning with Deep Generative Modelscited by 87
  11. 2021Generalized Multimodal ELBOcited by 87
  12. 2019Preventing Posterior Collapse with delta-VAEscited by 86
20 more
  1. 2016Modeling Missing Data in Clinical Time Series with RNNscited by 80
  2. 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
  3. 2021Representation learning for neural population activity with Neural Data Transformerscited by 50
  4. 2020Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAEcited by 48
  5. 2017Generative Models of Visually Grounded Imaginationcited by 44
  6. 2020Learning from Irregularly-Sampled Time Series: A Missing Data Perspectivecited by 40
  7. 2021Anomaly Detection of Time Series with Smoothness-Inducing Sequential Variational Auto-Encodercited by 39
  8. 2016Linear dynamical neural population models through nonlinear embeddingscited by 36
  9. 2015Black box variational inference for state space modelscited by 31
  10. 2016A State Space Approach for Piecewise-Linear Recurrent Neural Networks for Reconstructing Nonlinear Dynamics from Neural Measurementscited by 25
  11. 2022STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformercited by 23
  12. 2021Learning Multimodal VAEs through Mutual Supervisioncited by 18
  13. 2021Targeted Neural Dynamical Modelingcited by 18
  14. 2021Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Seriescited by 16
  15. 2019Neural Dynamics Discovery via Gaussian Process Recurrent Neural Networkscited by 13
  16. 2023Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Datacited by 9
  17. 2025BRAID: Input-Driven Nonlinear Dynamical Modeling of Neural-Behavioral Datacited by 8
  18. 2024eXponential FAmily Dynamical Systems (XFADS): Large-scale nonlinear Gaussian state-space modelingcited by 7
  19. 2021Deep inference of latent dynamics with spatio-temporal super-resolution using selective backpropagation through timecited by 6
  20. 2025Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Datacited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026Cross-Subject Modeling for Widefield Calcium Imaging via Atlas-Aligned Spatiotemporal Tokenizationcited by 1
  2. 2025Nonlinear Dynamical Modeling of Human Intracranial Brain Activity with Flexible Inferencecited by 0