# Neural system identification for large populations separating "what" and "where"

arXiv: https://arxiv.org/abs/1711.02653 · HTML: https://mlsanity.com/paper/1711.02653/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2017.

## Built on by (most cited first, 28 of 28)

- [The Neural Process Family: Survey, Applications and Perspectives](https://mlsanity.com/paper/2209.00517/) (2022)
- [Exploiting Noise as a Resource for Computation and Learning in Spiking Neural Networks](https://mlsanity.com/paper/2305.16044/) (2023)
- [The Sensorium competition on predicting large-scale mouse primary visual cortex activity](https://mlsanity.com/paper/2206.08666/) (2022)
- [A Spectral Theory of Neural Prediction and Alignment](https://mlsanity.com/paper/2309.12821/) (2023)
- [Transformer brain encoders explain human high-level visual responses](https://mlsanity.com/paper/2505.17329/) (2025)
- [Identifying Interpretable Visual Features in Artificial and Biological Neural Systems](https://mlsanity.com/paper/2310.11431/) (2023)
- [Deep Recurrent Encoder: A scalable end-to-end network to model brain signals](https://mlsanity.com/paper/2103.02339/) (2021)
- [V1T: large-scale mouse V1 response prediction using a Vision Transformer](https://mlsanity.com/paper/2302.03023/) (2023)
- [Deep Auto-encoder with Neural Response](https://mlsanity.com/paper/2111.15309/) (2021)
- [Transfer learning to decode brain states reflecting the relationship between cognitive tasks](https://mlsanity.com/paper/2206.03950/) (2022)
- [Adversarially trained neural representations may already be as robust as corresponding biological neural representations](https://mlsanity.com/paper/2206.11228/) (2022)
- [Modeling the Human Visual System: Comparative Insights from Response-Optimized and Task-Optimized Vision Models, Language Models, and different Readout Mechanisms](https://mlsanity.com/paper/2410.14031/) (2024)
- [Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex](https://mlsanity.com/paper/2505.15813/) (2025)
- [Understanding Neural Coding on Latent Manifolds by Sharing Features and Dividing Ensembles](https://mlsanity.com/paper/2210.03155/) (2022)
- [Most discriminative stimuli for functional cell type clustering](https://mlsanity.com/paper/2401.05342/) (2023)
- [Deep Continuous Networks](https://mlsanity.com/paper/2402.01557/) (2024)
- [Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors](https://mlsanity.com/paper/2410.16136/) (2024)
- [Meta-learning In-Context Enables Training-Free Cross Subject Brain Decoding](https://mlsanity.com/paper/2604.08537/) (2026)
- [Fine-Grained System Identification of Nonlinear Neural Circuits](https://mlsanity.com/paper/2106.05400/) (2021)
- [Self-Attention-Based Contextual Modulation Improves Neural System Identification](https://mlsanity.com/paper/2406.07843/) (2024)
- [Higher-Order Convolution Improves Neural Predictivity in the Retina](https://mlsanity.com/paper/2505.07620/) (2025)
- [Decomposing stimulus-specific sensory neural information via diffusion models](https://mlsanity.com/paper/2505.11309/) (2025)
- [Manifolds and Modules: How Function Develops in a Neural Foundation Model](https://mlsanity.com/paper/2512.07869/) (2025)
- [AVM: Towards Structure-Preserving Neural Response Modeling in the Visual Cortex Across Stimuli and Individuals](https://mlsanity.com/paper/2512.16948/) (2025)
- [NeuroFlow: Toward Unified Visual Encoding and Decoding from Neural Activity](https://mlsanity.com/paper/2604.09817/) (2026)
- [SIMON: Saliency-aware Integrative Multi-view Object-centric Neural Decoding](https://mlsanity.com/paper/2605.00401/) (2026)
- [STSBench: A Large-Scale Dataset for Modeling Neuronal Activity in the Dorsal Stream of Primate Visual Cortex](https://mlsanity.com/paper/2607.15631/) (2026)
- [Cross-attention encoding models reveal dynamic spatiotemporal routing across human higher visual cortex](https://mlsanity.com/paper/2609.36366/) (2026)

## Abstract (arXiv)

Neuroscientists classify neurons into different types that perform similar computations at different locations in the visual field. Traditional methods for neural system identification do not capitalize on this separation of 'what' and 'where'. Learning deep convolutional feature spaces that are shared among many neurons provides an exciting path forward, but the architectural design needs to account for data limitations: While new experimental techniques enable recordings from thousands of neurons, experimental time is limited so that one can sample only a small fraction of each neuron's response space. Here, we show that a major bottleneck for fitting convolutional neural networks (CNNs) to neural data is the estimation of the individual receptive field locations, a problem that has been scratched only at the surface thus far. We propose a CNN architecture with a sparse readout layer factorizing the spatial (where) and feature (what) dimensions. Our network scales well to thousands of neurons and short recordings and can be trained end-to-end. We evaluate this architecture on ground-truth data to explore the challenges and limitations of CNN-based system identification. Moreover, we show that our network model outperforms current state-of-the art system identification models of mouse primary visual cortex.