# The Uncanny Similarity of Recurrence and Depth

arXiv: https://arxiv.org/abs/2102.11011 · HTML: https://mlsanity.com/paper/2102.11011/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 22 Feb 2021.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [EMNIST: an extension of MNIST to handwritten letters](https://mlsanity.com/paper/1702.05373/) (2017)
- [Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation](https://mlsanity.com/paper/1802.06955/) (2018)
- [Residual Connections Encourage Iterative Inference](https://mlsanity.com/paper/1710.04773/) (2017)
- [Understanding Generalization through Visualizations](https://mlsanity.com/paper/1906.03291/) (2019)
- [Highway and Residual Networks learn Unrolled Iterative Estimation](https://mlsanity.com/paper/1612.07771/) (2016)
- [Trellis Networks for Sequence Modeling](https://mlsanity.com/paper/1810.06682/) (2018)
- [Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex](https://mlsanity.com/paper/1604.03640/) (2016)
- [Understanding Deep Architectures using a Recursive Convolutional Network](https://mlsanity.com/paper/1312.1847/) (2013)
- [Adversarial Attacks on Machine Learning Systems for High-Frequency Trading](https://mlsanity.com/paper/2002.09565/) (2020)
- [ShaResNet: reducing residual network parameter number by sharing weights](https://mlsanity.com/paper/1702.08782/) (2017)

## Built on by (most cited first, 9 of 9)

- [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022)
- [End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking](https://mlsanity.com/paper/2202.05826/) (2022)
- [A Review of Emerging Research Directions in Abstract Visual Reasoning](https://mlsanity.com/paper/2202.10284/) (2022)
- [CoBERL: Contrastive BERT for Reinforcement Learning](https://mlsanity.com/paper/2107.05431/) (2021)
- [Block-Recurrent Dynamics in Vision Transformers](https://mlsanity.com/paper/2512.19941/) (2025)
- [Short-Term Plasticity Neurons Learning to Learn and Forget](https://mlsanity.com/paper/2206.14048/) (2022)
- [Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels](https://mlsanity.com/paper/2311.06964/) (2023)
- [Traveling Waves Integrate Spatial Information Through Time](https://mlsanity.com/paper/2502.06034/) (2025)
- [Recurrent Sinusoidal INRs for Efficient High-Fidelity Representation](https://mlsanity.com/paper/2607.21485/) (2026)

## Abstract (arXiv)

It is widely believed that deep neural networks contain layer specialization, wherein neural networks extract hierarchical features representing edges and patterns in shallow layers and complete objects in deeper layers. Unlike common feed-forward models that have distinct filters at each layer, recurrent networks reuse the same parameters at various depths. In this work, we observe that recurrent models exhibit the same hierarchical behaviors and the same performance benefits with depth as feed-forward networks despite reusing the same filters at every recurrence. By training models of various feed-forward and recurrent architectures on several datasets for image classification as well as maze solving, we show that recurrent networks have the ability to closely emulate the behavior of non-recurrent deep models, often doing so with far fewer parameters.