# V1T: large-scale mouse V1 response prediction using a Vision Transformer

arXiv: https://arxiv.org/abs/2302.03023 · HTML: https://mlsanity.com/paper/2302.03023/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 85 references that are in the map.
- First posted on arXiv 6 Feb 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Vision Transformer for Small-Size Datasets](https://mlsanity.com/paper/2112.13492/) (2021)
- [Learnable latent embeddings for joint behavioral and neural analysis](https://mlsanity.com/paper/2204.00673/) (2022)
- [Are Convolutional Neural Networks or Transformers more like human vision?](https://mlsanity.com/paper/2105.07197/) (2021)
- [Cramming: Training a Language Model on a Single GPU in One Day](https://mlsanity.com/paper/2212.14034/) (2022)
- [Relating transformers to models and neural representations of the hippocampal formation](https://mlsanity.com/paper/2112.04035/) (2021)
- [Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition](https://mlsanity.com/paper/1406.3284/) (2014)
- [Representation learning for neural population activity with Neural Data Transformers](https://mlsanity.com/paper/2108.01210/) (2021)
- [Learning From Brains How to Regularize Machines](https://mlsanity.com/paper/1911.05072/) (2019)
- [Neural system identification for large populations separating "what" and "where"](https://mlsanity.com/paper/1711.02653/) (2017)
- [Towards robust vision by multi-task learning on monkey visual cortex](https://mlsanity.com/paper/2107.14344/) (2021)
- [STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer](https://mlsanity.com/paper/2206.04727/) (2022)
- [A rotation-equivariant convolutional neural network model of primary visual cortex](https://mlsanity.com/paper/1809.10504/) (2018)
- [The Sensorium competition on predicting large-scale mouse primary visual cortex activity](https://mlsanity.com/paper/2206.08666/) (2022)
- [Joint rotational invariance and adversarial training of a dual-stream Transformer yields state of the art Brain-Score for Area V4](https://mlsanity.com/paper/2203.06649/) (2022)
- [Neuronal Learning Analysis using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/2111.13073/) (2021)

## Built on by (most cited first, 6 of 6)

- [OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens](https://mlsanity.com/paper/2604.18827/) (2026)
- [Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors](https://mlsanity.com/paper/2410.16136/) (2024)
- [QuantFormer: Learning to Quantize for Neural Activity Forecasting in Mouse Visual Cortex](https://mlsanity.com/paper/2412.07264/) (2024)
- [Self-Attention-Based Contextual Modulation Improves Neural System Identification](https://mlsanity.com/paper/2406.07843/) (2024)
- [MEIcoder: Decoding Visual Stimuli from Neural Activity by Leveraging Most Exciting Inputs](https://mlsanity.com/paper/2510.20762/) (2025)
- [Manifolds and Modules: How Function Develops in a Neural Foundation Model](https://mlsanity.com/paper/2512.07869/) (2025)

## Abstract (arXiv)

Accurate predictive models of the visual cortex neural response to natural visual stimuli remain a challenge in computational neuroscience. In this work, we introduce V1T, a novel Vision Transformer based architecture that learns a shared visual and behavioral representation across animals. We evaluate our model on two large datasets recorded from mouse primary visual cortex and outperform previous convolution-based models by more than 12.7% in prediction performance. Moreover, we show that the self-attention weights learned by the Transformer correlate with the population receptive fields. Our model thus sets a new benchmark for neural response prediction and can be used jointly with behavioral and neural recordings to reveal meaningful characteristic features of the visual cortex.