# Understanding Neural Coding on Latent Manifolds by Sharing Features and Dividing Ensembles

arXiv: https://arxiv.org/abs/2210.03155 · HTML: https://mlsanity.com/paper/2210.03155/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 110 references that are in the map.
- First posted on arXiv 6 Oct 2022.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Hyperspherical Variational Auto-Encoders](https://mlsanity.com/paper/1804.00891/) (2018)
- [Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding](https://mlsanity.com/paper/2007.10930/) (2020)
- [Learnable latent embeddings for joint behavioral and neural analysis](https://mlsanity.com/paper/2204.00673/) (2022)
- [Towards the first adversarially robust neural network model on MNIST](https://mlsanity.com/paper/1805.09190/) (2018)
- [ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA](https://mlsanity.com/paper/2002.11537/) (2020)
- [Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity](https://mlsanity.com/paper/2109.04463/) (2021)
- [Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE](https://mlsanity.com/paper/2011.04798/) (2020)
- [Neural tuning and representational geometry](https://mlsanity.com/paper/2104.09743/) (2021)
- [Neural system identification for large populations separating "what" and "where"](https://mlsanity.com/paper/1711.02653/) (2017)
- [Towards robust vision by multi-task learning on monkey visual cortex](https://mlsanity.com/paper/2107.14344/) (2021)
- [Resisting Adversarial Attacks using Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1806.00081/) (2018)
- [A rotation-equivariant convolutional neural network model of primary visual cortex](https://mlsanity.com/paper/1809.10504/) (2018)
- [Manifold GPLVMs for discovering non-Euclidean latent structure in neural data](https://mlsanity.com/paper/2006.07429/) (2020)
- [The Sense of Place: Grid Cells in the Brain and the Transcendental Number e](https://mlsanity.com/paper/1304.0031/) (2013)
- [What can topology tell us about the neural code?](https://mlsanity.com/paper/1605.01905/) (2016)
- [Decoding of neural data using cohomological feature extraction](https://mlsanity.com/paper/1711.07205/) (2017)
- [Factorized Neural Processes for Neural Processes: K-Shot Prediction of Neural Responses](https://mlsanity.com/paper/2010.11810/) (2020)

## Built on by (most cited first, 3 of 3)

- [MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper](https://mlsanity.com/paper/2509.00996/) (2025)
- [A mathematical framework of intelligence and consciousness based on Riemannian Geometry](https://mlsanity.com/paper/2407.11024/) (2024)
- [Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors](https://mlsanity.com/paper/2410.16136/) (2024)

## Abstract (arXiv)

Systems neuroscience relies on two complementary views of neural data, characterized by single neuron tuning curves and analysis of population activity. These two perspectives combine elegantly in neural latent variable models that constrain the relationship between latent variables and neural activity, modeled by simple tuning curve functions. This has recently been demonstrated using Gaussian processes, with applications to realistic and topologically relevant latent manifolds. Those and previous models, however, missed crucial shared coding properties of neural populations. We propose feature sharing across neural tuning curves which significantly improves performance and helps optimization. We also propose a solution to the ensemble detection problem, where different groups of neurons, i.e., ensembles, can be modulated by different latent manifolds. Achieved through a soft clustering of neurons during training, this allows for the separation of mixed neural populations in an unsupervised manner. These innovations lead to more interpretable models of neural population activity that train well and perform better even on mixtures of complex latent manifolds. Finally, we apply our method on a recently published grid cell dataset, and recover distinct ensembles, infer toroidal latents and predict neural tuning curves in a single integrated modeling framework.