# Who invented deep residual learning?

arXiv: https://arxiv.org/abs/2509.24732 · HTML: https://mlsanity.com/paper/2509.24732/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 49 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [Random Feature Attention](https://mlsanity.com/paper/2103.02143/) (2021)
- [Hypercolumns for Object Segmentation and Fine-grained Localization](https://mlsanity.com/paper/1411.5752/) (2014)
- [Going Beyond Linear Transformers with Recurrent Fast Weight Programmers](https://mlsanity.com/paper/2106.06295/) (2021)
- [The Importance of Being Recurrent for Modeling Hierarchical Structure](https://mlsanity.com/paper/1803.03585/) (2018)
- [Learning to Reason with Third-Order Tensor Products](https://mlsanity.com/paper/1811.12143/) (2018)
- [Annotated History of Modern AI and Deep Learning](https://mlsanity.com/paper/2212.11279/) (2022)
- [Exploring the Promise and Limits of Real-Time Recurrent Learning](https://mlsanity.com/paper/2305.19044/) (2023)
- [Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules](https://mlsanity.com/paper/2206.01649/) (2022)
- [Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules](https://mlsanity.com/paper/2210.06184/) (2022)

## Abstract (arXiv)

Modern AI is based on deep artificial neural networks (NNs). As of 2025, the most cited scientific article of the 21st century is an NN paper on deep residual learning with residual connections. Who invented this? We present a timeline of the evolution of deep residual learning.