# Deep Learning Models of the Retinal Response to Natural Scenes

arXiv: https://arxiv.org/abs/1702.01825 · HTML: https://mlsanity.com/paper/1702.01825/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2017.

## Built on by (most cited first, 30 of 36)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [Disentanglement with Biological Constraints: A Theory of Functional Cell Types](https://mlsanity.com/paper/2210.01768/) (2022)
- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [Spiking Vision Transformer with Saccadic Attention](https://mlsanity.com/paper/2502.12677/) (2025)
- [Exploiting Noise as a Resource for Computation and Learning in Spiking Neural Networks](https://mlsanity.com/paper/2305.16044/) (2023)
- [The Sensorium competition on predicting large-scale mouse primary visual cortex activity](https://mlsanity.com/paper/2206.08666/) (2022)
- [Brain-Model Evaluations Need the NeuroAI Turing Test](https://mlsanity.com/paper/2502.16238/) (2025)
- [Better artificial intelligence does not mean better models of biology](https://mlsanity.com/paper/2504.16940/) (2025)
- [Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks](https://mlsanity.com/paper/2109.15089/) (2021)
- [Fitting summary statistics of neural data with a differentiable spiking network simulator](https://mlsanity.com/paper/2106.10064/) (2021)
- [Deep Auto-encoder with Neural Response](https://mlsanity.com/paper/2111.15309/) (2021)
- [Temporal Conditioning Spiking Latent Variable Models of the Neural Response to Natural Visual Scenes](https://mlsanity.com/paper/2306.12045/) (2023)
- [Unraveling the geometry of visual relational reasoning](https://mlsanity.com/paper/2502.17382/) (2025)
- [Most discriminative stimuli for functional cell type clustering](https://mlsanity.com/paper/2401.05342/) (2023)
- [Neural Ganglion Sensors: Learning Task-specific Event Cameras Inspired by the Neural Circuit of the Human Retina](https://mlsanity.com/paper/2504.13457/) (2025)
- [Machine Learning Method for Functional Assessment of Retinal Models](https://mlsanity.com/paper/2202.02443/) (2022)
- [Signal Processing in the Retina: Interpretable Graph Classifier to Predict Ganglion Cell Responses](https://mlsanity.com/paper/2401.01813/) (2024)
- [Explicitly Modeling Pre-Cortical Vision with a Neuro-Inspired Front-End Improves CNN Robustness](https://mlsanity.com/paper/2409.16838/) (2024)
- [Fine-Grained System Identification of Nonlinear Neural Circuits](https://mlsanity.com/paper/2106.05400/) (2021)
- [Variational framework for partially-measured physical system control: examples of vision neuroscience and optical random media](https://mlsanity.com/paper/2110.13228/) (2021)
- [Integrating Statistical and Machine Learning Approaches to Identify Receptive Field Structure in Neural Populations](https://mlsanity.com/paper/2208.12025/) (2022)
- [Scaling and Resizing Symmetry in Feedforward Networks](https://mlsanity.com/paper/2306.15015/) (2023)
- [Visual attention information can be traced on cortical response but not on the retina: evidence from electrophysiological mouse data using natural images as stimuli](https://mlsanity.com/paper/2308.00526/) (2023)
- [Exploring Biological Neuronal Correlations with Quantum Generative Models](https://mlsanity.com/paper/2409.09125/) (2024)
- [Higher-Order Convolution Improves Neural Predictivity in the Retina](https://mlsanity.com/paper/2505.07620/) (2025)
- [MEIcoder: Decoding Visual Stimuli from Neural Activity by Leveraging Most Exciting Inputs](https://mlsanity.com/paper/2510.20762/) (2025)
- [Language models as tools for investigating the distinction between possible and impossible natural languages](https://mlsanity.com/paper/2512.09394/) (2025)
- [Large language models have learned to use language](https://mlsanity.com/paper/2512.12447/) (2025)
- [Large language models are not about natural language](https://mlsanity.com/paper/2512.13441/) (2025)

## Abstract (arXiv)

A central challenge in neuroscience is to understand neural computations and circuit mechanisms that underlie the encoding of ethologically relevant, natural stimuli. In multilayered neural circuits, nonlinear processes such as synaptic transmission and spiking dynamics present a significant obstacle to the creation of accurate computational models of responses to natural stimuli. Here we demonstrate that deep convolutional neural networks (CNNs) capture retinal responses to natural scenes nearly to within the variability of a cell's response, and are markedly more accurate than linear-nonlinear (LN) models and Generalized Linear Models (GLMs). Moreover, we find two additional surprising properties of CNNs: they are less susceptible to overfitting than their LN counterparts when trained on small amounts of data, and generalize better when tested on stimuli drawn from a different distribution (e.g. between natural scenes and white noise). Examination of trained CNNs reveals several properties. First, a richer set of feature maps is necessary for predicting the responses to natural scenes compared to white noise. Second, temporally precise responses to slowly varying inputs originate from feedforward inhibition, similar to known retinal mechanisms. Third, the injection of latent noise sources in intermediate layers enables our model to capture the sub-Poisson spiking variability observed in retinal ganglion cells. Fourth, augmenting our CNNs with recurrent lateral connections enables them to capture contrast adaptation as an emergent property of accurately describing retinal responses to natural scenes. These methods can be readily generalized to other sensory modalities and stimulus ensembles. Overall, this work demonstrates that CNNs not only accurately capture sensory circuit responses to natural scenes, but also yield information about the circuit's internal structure and function.