# A neural network trained to predict future video frames mimics critical properties of biological neuronal responses and perception

arXiv: https://arxiv.org/abs/1805.10734 · HTML: https://mlsanity.com/paper/1805.10734/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2018.

## Built on by (most cited first, 6 of 6)

- [Mixed Evidence for Gestalt Grouping in Deep Neural Networks](https://mlsanity.com/paper/2203.07302/) (2022)
- [Predictive coding feedback results in perceived illusory contours in a recurrent neural network](https://mlsanity.com/paper/2102.01955/) (2021)
- [Using Human Perception to Regularize Transfer Learning](https://mlsanity.com/paper/2211.07885/) (2022)
- [Abutting Grating Illusion: Cognitive Challenge to Neural Network Models](https://mlsanity.com/paper/2208.03958/) (2022)
- [Unsupervised learning of features and object boundaries from local prediction](https://mlsanity.com/paper/2205.14195/) (2022)
- [A Review of Mechanistic Models of Event Comprehension](https://mlsanity.com/paper/2409.18992/) (2024)

## Abstract (arXiv)

While deep neural networks take loose inspiration from neuroscience, it is an open question how seriously to take the analogies between artificial deep networks and biological neuronal systems. Interestingly, recent work has shown that deep convolutional neural networks (CNNs) trained on large-scale image recognition tasks can serve as strikingly good models for predicting the responses of neurons in visual cortex to visual stimuli, suggesting that analogies between artificial and biological neural networks may be more than superficial. However, while CNNs capture key properties of the average responses of cortical neurons, they fail to explain other properties of these neurons. For one, CNNs typically require large quantities of labeled input data for training. Our own brains, in contrast, rarely have access to this kind of supervision, so to the extent that representations are similar between CNNs and brains, this similarity must arise via different training paths. In addition, neurons in visual cortex produce complex time-varying responses even to static inputs, and they dynamically tune themselves to temporal regularities in the visual environment. We argue that these differences are clues to fundamental differences between the computations performed in the brain and in deep networks. To begin to close the gap, here we study the emergent properties of a previously-described recurrent generative network that is trained to predict future video frames in a self-supervised manner. Remarkably, the model is able to capture a wide variety of seemingly disparate phenomena observed in visual cortex, ranging from single unit response dynamics to complex perceptual motion illusions. These results suggest potentially deep connections between recurrent predictive neural network models and the brain, providing new leads that can enrich both fields.