# A Selective Overview of Deep Learning

arXiv: https://arxiv.org/abs/1904.05526 · HTML: https://mlsanity.com/paper/1904.05526/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2019.

## Built on by (most cited first, 30 of 31)

- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [The Power of Contrast for Feature Learning: A Theoretical Analysis](https://mlsanity.com/paper/2110.02473/) (2021)
- [A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models](https://mlsanity.com/paper/2401.07187/) (2024)
- [Theoretical analysis of deep neural networks for temporally dependent observations](https://mlsanity.com/paper/2210.11530/) (2022)
- [A Probabilistic Approach to Neural Network Pruning](https://mlsanity.com/paper/2105.10065/) (2021)
- [Statistical Deep Learning for Spatial and Spatio-Temporal Data](https://mlsanity.com/paper/2206.02218/) (2022)
- [A CMDP-within-online framework for Meta-Safe Reinforcement Learning](https://mlsanity.com/paper/2405.16601/) (2024)
- [Restricted Strong Convexity of Deep Learning Models with Smooth Activations](https://mlsanity.com/paper/2209.15106/) (2022)
- [Non-Asymptotic Guarantees for Robust Statistical Learning under Infinite Variance Assumption](https://mlsanity.com/paper/2201.03182/) (2022)
- [A systematic review of biologically-informed deep learning models for cancer: fundamental trends for encoding and interpreting oncology data](https://mlsanity.com/paper/2207.00812/) (2022)
- [Merging Two Cultures: Deep and Statistical Learning](https://mlsanity.com/paper/2110.11561/) (2021)
- [Intrinsic and extrinsic deep learning on manifolds](https://mlsanity.com/paper/2302.08606/) (2023)
- [Screening for Diabetes Mellitus in the U.S. Population Using Neural Network Models and Complex Survey Designs](https://mlsanity.com/paper/2403.19752/) (2024)
- [Deep learning with missing data](https://mlsanity.com/paper/2504.15388/) (2025)
- [Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks](https://mlsanity.com/paper/2111.12963/) (2021)
- [Robust Deep Neural Network Estimation for Multi-dimensional Functional Data](https://mlsanity.com/paper/2205.09604/) (2022)
- [Blind Estimation of a Doubly Selective OFDM Channel: A Deep Learning Algorithm and Theory](https://mlsanity.com/paper/2206.07483/) (2022)
- [Subject clustering by IF-PCA and several recent methods](https://mlsanity.com/paper/2306.05363/) (2023)
- [Stringological sequence prediction I: efficient algorithms for predicting highly repetitive sequences](https://mlsanity.com/paper/2603.26852/) (2026)
- [Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors](https://mlsanity.com/paper/2605.27967/) (2026)
- [Individualized Risk Assessment of Preoperative Opioid Use by Interpretable Neural Network Regression](https://mlsanity.com/paper/2205.08370/) (2022)
- [Spectral Gap Regularization of Neural Networks](https://mlsanity.com/paper/2304.03096/) (2023)
- [Deep Neural Networks for Semiparametric Frailty Models via H-likelihood](https://mlsanity.com/paper/2307.06581/) (2023)
- [Heterogeneous virus classification using a functional deep learning model based on transmission electron microscopy images (Preprint)](https://mlsanity.com/paper/2405.15563/) (2024)
- [Generative artificial intelligence in dentistry: Current approaches and future challenges](https://mlsanity.com/paper/2407.17532/) (2024)
- [Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theories](https://mlsanity.com/paper/2412.18756/) (2024)
- [Boosting Statistic Learning with Synthetic Data from Pretrained Large Models](https://mlsanity.com/paper/2505.04992/) (2025)
- [Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networks](https://mlsanity.com/paper/2507.04779/) (2025)
- [Single Pixel Image Classification using an Ultrafast Digital Light Projector](https://mlsanity.com/paper/2603.12036/) (2026)

## Abstract (arXiv)

Deep learning has arguably achieved tremendous success in recent years. In simple words, deep learning uses the composition of many nonlinear functions to model the complex dependency between input features and labels. While neural networks have a long history, recent advances have greatly improved their performance in computer vision, natural language processing, etc. From the statistical and scientific perspective, it is natural to ask: What is deep learning? What are the new characteristics of deep learning, compared with classical methods? What are the theoretical foundations of deep learning? To answer these questions, we introduce common neural network models (e.g., convolutional neural nets, recurrent neural nets, generative adversarial nets) and training techniques (e.g., stochastic gradient descent, dropout, batch normalization) from a statistical point of view. Along the way, we highlight new characteristics of deep learning (including depth and over-parametrization) and explain their practical and theoretical benefits. We also sample recent results on theories of deep learning, many of which are only suggestive. While a complete understanding of deep learning remains elusive, we hope that our perspectives and discussions serve as a stimulus for new statistical research.