# When predict can also explain: few-shot prediction to select better neural latents

arXiv: https://arxiv.org/abs/2405.14425 · HTML: https://mlsanity.com/paper/2405.14425/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 74 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [Transformers without Tears: Improving the Normalization of Self-Attention](https://mlsanity.com/paper/1910.05895/) (2019)
- [Enhancing Computational Fluid Dynamics with Machine Learning](https://mlsanity.com/paper/2110.02085/) (2021)
- [Universality and individuality in neural dynamics across large populations of recurrent networks](https://mlsanity.com/paper/1907.08549/) (2019)
- [Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity](https://mlsanity.com/paper/2109.04463/) (2021)
- [Representation learning for neural population activity with Neural Data Transformers](https://mlsanity.com/paper/2108.01210/) (2021)
- [Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI](https://mlsanity.com/paper/1902.07186/) (2019)
- [STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer](https://mlsanity.com/paper/2206.04727/) (2022)
- [Variational Latent Gaussian Process for Recovering Single-Trial Dynamics from Population Spike Trains](https://mlsanity.com/paper/1604.03053/) (2016)
- [Neural Dynamics Discovery via Gaussian Process Recurrent Neural Networks](https://mlsanity.com/paper/1907.00650/) (2019)
- [lfads-torch: A modular and extensible implementation of latent factor analysis via dynamical systems](https://mlsanity.com/paper/2309.01230/) (2023)
- [Expressive architectures enhance interpretability of dynamics-based neural population models](https://mlsanity.com/paper/2212.03771/) (2022)
- [Expressive dynamics models with nonlinear injective readouts enable reliable recovery of latent features from neural activity](https://mlsanity.com/paper/2309.06402/) (2023)
- [Inferring Population Dynamics in Macaque Cortex](https://mlsanity.com/paper/2304.06040/) (2023)

## Abstract (arXiv)

Latent variable models serve as powerful tools to infer underlying dynamics from observed neural activity. Ideally, the inferred dynamics should align with true ones. However, due to the absence of ground truth data, prediction benchmarks are often employed as proxies. One widely-used method, textitco-smoothing, involves jointly estimating latent variables and predicting observations along held-out channels to assess model performance. In this study, we reveal the limitations of the co-smoothing prediction framework and propose a remedy. Using a student-teacher setup, we demonstrate that models with high co-smoothing can have arbitrary extraneous dynamics in their latent representations. To address this, we introduce a secondary metric -- textitfew-shot co-smoothing, performing regression from the latent variables to held-out neurons in the data using fewer trials. Our results indicate that among models with near-optimal co-smoothing, those with extraneous dynamics underperform in the few-shot co-smoothing compared to `minimal' models that are devoid of such dynamics. We provide analytical insights into the origin of this phenomenon and further validate our findings on four standard neural datasets using a state-of-the-art method: STNDT. In the absence of ground truth, we suggest a novel measure to validate our approach. By cross-decoding the latent variables of all model pairs with high co-smoothing, we identify models with minimal extraneous dynamics. We find a correlation between few-shot co-smoothing performance and this new measure. In summary, we present a novel prediction metric designed to yield latent variables that more accurately reflect the ground truth, offering a significant improvement for latent dynamics inference.