# Recent advances in deep learning theory

arXiv: https://arxiv.org/abs/2012.10931 · HTML: https://mlsanity.com/paper/2012.10931/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2020.

## Built on by (most cited first, 30 of 41)

- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training](https://mlsanity.com/paper/2206.00187/) (2022)
- [A Survey on Kolmogorov-Arnold Network](https://mlsanity.com/paper/2411.06078/) (2024)
- [FIBA: Frequency-Injection based Backdoor Attack in Medical Image Analysis](https://mlsanity.com/paper/2112.01148/) (2021)
- [Exploring Sequence Feature Alignment for Domain Adaptive Detection Transformers](https://mlsanity.com/paper/2107.12636/) (2021)
- [Strength of Minibatch Noise in SGD](https://mlsanity.com/paper/2102.05375/) (2021)
- [Topology-aware Generalization of Decentralized SGD](https://mlsanity.com/paper/2206.12680/) (2022)
- [Robust Unlearnable Examples: Protecting Data Against Adversarial Learning](https://mlsanity.com/paper/2203.14533/) (2022)
- [Temporal Domain Generalization with Drift-Aware Dynamic Neural Networks](https://mlsanity.com/paper/2205.10664/) (2022)
- [E(2)-Equivariant Vision Transformer](https://mlsanity.com/paper/2306.06722/) (2023)
- [Knowledge Removal in Sampling-based Bayesian Inference](https://mlsanity.com/paper/2203.12964/) (2022)
- [A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models](https://mlsanity.com/paper/2401.07187/) (2024)
- [Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squares](https://mlsanity.com/paper/2206.01274/) (2022)
- [Benefits of Permutation-Equivariance in Auction Mechanisms](https://mlsanity.com/paper/2210.05579/) (2022)
- [ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmaps](https://mlsanity.com/paper/2305.06472/) (2023)
- [Cutting-edge 3D Medical Image Segmentation Methods in 2020: Are Happy Families All Alike?](https://mlsanity.com/paper/2101.00232/) (2021)
- [Bayesian Inference Forgetting](https://mlsanity.com/paper/2101.06417/) (2021)
- [Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations](https://mlsanity.com/paper/2102.13233/) (2021)
- [Self-Ensembling GAN for Cross-Domain Semantic Segmentation](https://mlsanity.com/paper/2112.07999/) (2021)
- [Understanding Deep Learning via Decision Boundary](https://mlsanity.com/paper/2206.01515/) (2022)
- [Why Quantization Improves Generalization: NTK of Binary Weight Neural Networks](https://mlsanity.com/paper/2206.05916/) (2022)
- [Neural networks behave as hash encoders: An empirical study](https://mlsanity.com/paper/2101.05490/) (2021)
- [VITA: A Multi-Source Vicinal Transfer Augmentation Method for Out-of-Distribution Generalization](https://mlsanity.com/paper/2204.11531/) (2022)
- [Domain Generalization without Excess Empirical Risk](https://mlsanity.com/paper/2308.15856/) (2023)
- [Generalization Bounds in Hybrid Quantum-Classical Machine Learning Models](https://mlsanity.com/paper/2504.08456/) (2025)
- [Advancing biological super-resolution microscopy through deep learning: a brief review](https://mlsanity.com/paper/2106.13064/) (2021)
- [Less is More -- Towards parsimonious multi-task models using structured sparsity](https://mlsanity.com/paper/2308.12114/) (2023)
- [Class-wise Generalization Error: an Information-Theoretic Analysis](https://mlsanity.com/paper/2401.02904/) (2024)
- [TITAN: Query-Token based Domain Adaptive Adversarial Learning](https://mlsanity.com/paper/2506.21484/) (2025)

## Abstract (arXiv)

Deep learning is usually described as an experiment-driven field under continuous criticizes of lacking theoretical foundations. This problem has been partially fixed by a large volume of literature which has so far not been well organized. This paper reviews and organizes the recent advances in deep learning theory. The literature is categorized in six groups: (1) complexity and capacity-based approaches for analyzing the generalizability of deep learning; (2) stochastic differential equations and their dynamic systems for modelling stochastic gradient descent and its variants, which characterize the optimization and generalization of deep learning, partially inspired by Bayesian inference; (3) the geometrical structures of the loss landscape that drives the trajectories of the dynamic systems; (4) the roles of over-parameterization of deep neural networks from both positive and negative perspectives; (5) theoretical foundations of several special structures in network architectures; and (6) the increasingly intensive concerns in ethics and security and their relationships with generalizability.