# Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex

arXiv: https://arxiv.org/abs/1604.03640 · HTML: https://mlsanity.com/paper/1604.03640/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2016.

## Built on by (most cited first, 30 of 42)

- [Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks](https://mlsanity.com/paper/2106.04537/) (2021)
- [Neural Flows: Efficient Alternative to Neural ODEs](https://mlsanity.com/paper/2110.13040/) (2021)
- [Encoding physics to learn reaction-diffusion processes](https://mlsanity.com/paper/2106.04781/) (2021)
- [Adaptive Rational Activations to Boost Deep Reinforcement Learning](https://mlsanity.com/paper/2102.09407/) (2021)
- [Block-Recurrent Dynamics in Vision Transformers](https://mlsanity.com/paper/2512.19941/) (2025)
- [Banana: Banach Fixed-Point Network for Pointcloud Segmentation with Inter-Part Equivariance](https://mlsanity.com/paper/2305.16314/) (2023)
- [The least-control principle for local learning at equilibrium](https://mlsanity.com/paper/2207.01332/) (2022)
- [Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment](https://mlsanity.com/paper/2312.09257/) (2023)
- [Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation](https://mlsanity.com/paper/2110.11852/) (2021)
- [Hierarchical Reasoning Models: Perspectives and Misconceptions](https://mlsanity.com/paper/2510.00355/) (2025)
- [Are Deep Neural Networks Adequate Behavioural Models of Human Visual Perception?](https://mlsanity.com/paper/2305.17023/) (2023)
- [Better artificial intelligence does not mean better models of biology](https://mlsanity.com/paper/2504.16940/) (2025)
- [The Uncanny Similarity of Recurrence and Depth](https://mlsanity.com/paper/2102.11011/) (2021)
- [An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits](https://mlsanity.com/paper/2212.10744/) (2022)
- [System identification of neural systems: If we got it right, would we know?](https://mlsanity.com/paper/2302.06677/) (2023)
- [Hard Encoding of Physics for Learning Spatiotemporal Dynamics](https://mlsanity.com/paper/2105.00557/) (2021)
- [Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels](https://mlsanity.com/paper/2311.06964/) (2023)
- [TDAM: Top-Down Attention Module for Contextually Guided Feature Selection in CNNs](https://mlsanity.com/paper/2111.13470/) (2021)
- [ThreshNet: An Efficient DenseNet Using Threshold Mechanism to Reduce Connections](https://mlsanity.com/paper/2201.03013/) (2022)
- [Geometric and Topological Inference for Deep Representations of Complex Networks](https://mlsanity.com/paper/2203.05488/) (2022)
- [Entangled Residual Mappings](https://mlsanity.com/paper/2206.01261/) (2022)
- [Understanding Adversarial Robustness of Vision Transformers via Cauchy Problem](https://mlsanity.com/paper/2208.00906/) (2022)
- [Rethinking ResNets: Improved Stacking Strategies With High Order Schemes](https://mlsanity.com/paper/2103.15244/) (2021)
- [Category-orthogonal object features guide information processing in recurrent neural networks trained for object categorization](https://mlsanity.com/paper/2111.07898/) (2021)
- [Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks](https://mlsanity.com/paper/2111.12330/) (2021)
- [The Mori-Zwanzig formulation of deep learning](https://mlsanity.com/paper/2209.05544/) (2022)
- [Multicoated and Folded Graph Neural Networks with Strong Lottery Tickets](https://mlsanity.com/paper/2312.03236/) (2023)
- [Deep Continuous Networks](https://mlsanity.com/paper/2402.01557/) (2024)
- [The Ikshana Hypothesis of Human Scene Understanding](https://mlsanity.com/paper/2101.10837/) (2021)
- [A Goal-Driven Approach to Systems Neuroscience](https://mlsanity.com/paper/2311.02704/) (2023)

## Abstract (arXiv)

We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex. We begin with the observation that a special type of shallow RNN is exactly equivalent to a very deep ResNet with weight sharing among the layers. A direct implementation of such a RNN, although having orders of magnitude fewer parameters, leads to a performance similar to the corresponding ResNet. We propose 1) a generalization of both RNN and ResNet architectures and 2) the conjecture that a class of moderately deep RNNs is a biologically-plausible model of the ventral stream in visual cortex. We demonstrate the effectiveness of the architectures by testing them on the CIFAR-10 and ImageNet dataset.