# On the Origin of Deep Learning

arXiv: https://arxiv.org/abs/1702.07800 · HTML: https://mlsanity.com/paper/1702.07800/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2017.

## Built on by (most cited first, 23 of 23)

- [Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges](https://mlsanity.com/paper/2102.01168/) (2021)
- [Neurosymbolic Reinforcement Learning and Planning: A Survey](https://mlsanity.com/paper/2309.01038/) (2023)
- [Automated Algorithm Selection: from Feature-Based to Feature-Free Approaches](https://mlsanity.com/paper/2203.13392/) (2022)
- [A Comprehensive Review of Community Detection in Graphs](https://mlsanity.com/paper/2309.11798/) (2023)
- [Toward Learning Robust and Invariant Representations with Alignment Regularization and Data Augmentation](https://mlsanity.com/paper/2206.01909/) (2022)
- [Knowledge Engineering using Large Language Models](https://mlsanity.com/paper/2310.00637/) (2023)
- [Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency](https://mlsanity.com/paper/2402.19366/) (2024)
- [The Power of Communication in a Distributed Multi-Agent System](https://mlsanity.com/paper/2111.15611/) (2021)
- [Collaborative Auto-Curricula Multi-Agent Reinforcement Learning with Graph Neural Network Communication Layer for Open-ended Wildfire-Management Resource Distribution](https://mlsanity.com/paper/2204.11350/) (2022)
- [Foundation models in brief: A historical, socio-technical focus](https://mlsanity.com/paper/2212.08967/) (2022)
- [Mitigating Confounding in Speech-Based Dementia Detection through Weight Masking](https://mlsanity.com/paper/2506.05610/) (2025)
- [ganX -- generate artificially new XRF a python library to generate MA-XRF raw data out of RGB images](https://mlsanity.com/paper/2304.14078/) (2023)
- [Identifying and Consolidating Knowledge Engineering Requirements](https://mlsanity.com/paper/2306.15124/) (2023)
- [Deep Learning in Computational Biology: Advancements, Challenges, and Future Outlook](https://mlsanity.com/paper/2310.03086/) (2023)
- [Emulating Spatio-Temporal Realizations of Three-Dimensional Isotropic Turbulence via Deep Sequence Learning Models](https://mlsanity.com/paper/2112.03469/) (2021)
- [Parkinson's Disease Diagnosis Using Deep Learning](https://mlsanity.com/paper/2101.05631/) (2021)
- [OpTorch: Optimized deep learning architectures for resource limited environments](https://mlsanity.com/paper/2105.00619/) (2021)
- [A deep neural network for multi-species fish detection using multiple acoustic cameras](https://mlsanity.com/paper/2109.10664/) (2021)
- [Analysis and prediction of heart stroke from ejection fraction and serum creatinine using LSTM deep learning approach](https://mlsanity.com/paper/2209.13799/) (2022)
- [A Deep Learning Architecture for Passive Microwave Precipitation Retrievals using CloudSat and GPM Data](https://mlsanity.com/paper/2212.02236/) (2022)
- [Learning to Communicate and Collaborate in a Competitive Multi-Agent Setup to Clean the Ocean from Macroplastics](https://mlsanity.com/paper/2304.05872/) (2023)
- [A Bibliographic Study on Artificial Intelligence Research: Global Panorama and Indian Appearance](https://mlsanity.com/paper/2308.00705/) (2023)
- [Hopfield Networks Meet Big Data: A Brain-Inspired Deep Learning Framework for Semantic Data Linking](https://mlsanity.com/paper/2503.03084/) (2025)

## Abstract (arXiv)

This paper is a review of the evolutionary history of deep learning models. It covers from the genesis of neural networks when associationism modeling of the brain is studied, to the models that dominate the last decade of research in deep learning like convolutional neural networks, deep belief networks, and recurrent neural networks. In addition to a review of these models, this paper primarily focuses on the precedents of the models above, examining how the initial ideas are assembled to construct the early models and how these preliminary models are developed into their current forms. Many of these evolutionary paths last more than half a century and have a diversity of directions. For example, CNN is built on prior knowledge of biological vision system; DBN is evolved from a trade-off of modeling power and computation complexity of graphical models and many nowadays models are neural counterparts of ancient linear models. This paper reviews these evolutionary paths and offers a concise thought flow of how these models are developed, and aims to provide a thorough background for deep learning. More importantly, along with the path, this paper summarizes the gist behind these milestones and proposes many directions to guide the future research of deep learning.