mlsanityData to 14 Oct 2026

The Uncanny Similarity of Recurrence and Depth

arXiv 2102.11011 ↗First posted 22 Feb 2021cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

It is widely believed that deep neural networks contain layer specialization, wherein neural networks extract hierarchical features representing edges and patterns in shallow layers and complete objects in deeper layers. Unlike common feed-forward models that have distinct filters at each layer, recurrent networks reuse the same parameters at various depths. In this work, we observe that recurrent models exhibit the same hierarchical behaviors and the same performance benefits with depth as feed-forward networks despite reusing the same filters at every recurrence. By training models of various feed-forward and recurrent architectures on several datasets for image classification as well as maze solving, we show that recurrent networks have the ability to closely emulate the behavior of non-recurrent deep models, often doing so with far fewer parameters.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

24 of 35 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2018Neural Ordinary Differential Equationscited by 3,779
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  8. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  9. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  11. 2019Deep Equilibrium Modelscited by 615
  12. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
12 more
  1. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  2. 2015Understanding Neural Networks Through Deep Visualizationcited by 334
  3. 2017EMNIST: an extension of MNIST to handwritten letterscited by 213
  4. 2018Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentationcited by 180
  5. 2017Residual Connections Encourage Iterative Inferencecited by 77
  6. 2019Understanding Generalization through Visualizationscited by 54
  7. 2016Highway and Residual Networks learn Unrolled Iterative Estimationcited by 52
  8. 2018Trellis Networks for Sequence Modelingcited by 52
  9. 2016Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortexcited by 42
  10. 2013Understanding Deep Architectures using a Recursive Convolutional Networkcited by 11
  11. 2020Adversarial Attacks on Machine Learning Systems for High-Frequency Tradingcited by 11
  12. 2017ShaResNet: reducing residual network parameter number by sharing weightscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 1; 2024: 0; 2025: 2; 2026: 1.

9 papers.

  1. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  2. 2022End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinkingcited by 33
  3. 2022A Review of Emerging Research Directions in Abstract Visual Reasoningcited by 33
  4. 2021CoBERL: Contrastive BERT for Reinforcement Learningcited by 21
  5. 2025Block-Recurrent Dynamics in Vision Transformerscited by 20
  6. 2022Short-Term Plasticity Neurons Learning to Learn and Forgetcited by 6
  7. 2023Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levelscited by 6
  8. 2025Traveling Waves Integrate Spatial Information Through Timecited by 0
  9. 2026Recurrent Sinusoidal INRs for Efficient High-Fidelity Representationcited by 0