mlsanityData to 7 Oct 2026

Recent advances in deep learning theory

arXiv 2012.10931 ↗First posted 20 Dec 2020cs.LG stat.MLcited by 41

Not on the map. Why?

Abstract (arXiv)

Deep learning is usually described as an experiment-driven field under continuous criticizes of lacking theoretical foundations. This problem has been partially fixed by a large volume of literature which has so far not been well organized. This paper reviews and organizes the recent advances in deep learning theory. The literature is categorized in six groups: (1) complexity and capacity-based approaches for analyzing the generalizability of deep learning; (2) stochastic differential equations and their dynamic systems for modelling stochastic gradient descent and its variants, which characterize the optimization and generalization of deep learning, partially inspired by Bayesian inference; (3) the geometrical structures of the loss landscape that drives the trajectories of the dynamic systems; (4) the roles of over-parameterization of deep neural networks from both positive and negative perspectives; (5) theoretical foundations of several special structures in network architectures; and (6) the increasingly intensive concerns in ethics and security and their relationships with generalizability.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 12; 2023: 6; 2024: 3; 2025: 7; 2026: 1.

41 papers.

Sort:

By area:

  1. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  2. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  3. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  4. 2024A Survey on Kolmogorov-Arnold Networkcited by 67
  5. 2021FIBA: Frequency-Injection based Backdoor Attack in Medical Image Analysiscited by 60
  6. 2021Exploring Sequence Feature Alignment for Domain Adaptive Detection Transformerscited by 49
  7. 2021Strength of Minibatch Noise in SGDcited by 39
  8. 2022Topology-aware Generalization of Decentralized SGDcited by 37
  9. 2022Robust Unlearnable Examples: Protecting Data Against Adversarial Learningcited by 34
  10. 2022Temporal Domain Generalization with Drift-Aware Dynamic Neural Networkscited by 32
  11. 2023E(2)-Equivariant Vision Transformercited by 27
  12. 2022Knowledge Removal in Sampling-based Bayesian Inferencecited by 20
29 more
  1. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  2. 2022Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squarescited by 12
  3. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  4. 2021Cutting-edge 3D Medical Image Segmentation Methods in 2020: Are Happy Families All Alike?cited by 7
  5. 2021Bayesian Inference Forgettingcited by 7
  6. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  7. 2021Self-Ensembling GAN for Cross-Domain Semantic Segmentationcited by 7
  8. 2022Understanding Deep Learning via Decision Boundarycited by 7
  9. 2022Benefits of Permutation-Equivariance in Auction Mechanismscited by 7
  10. 2022Why Quantization Improves Generalization: NTK of Binary Weight Neural Networkscited by 5
  11. 2021Neural networks behave as hash encoders: An empirical studycited by 3
  12. 2022VITA: A Multi-Source Vicinal Transfer Augmentation Method for Out-of-Distribution Generalizationcited by 3
  13. 2023Domain Generalization without Excess Empirical Riskcited by 3
  14. 2025Generalization Bounds in Hybrid Quantum-Classical Machine Learning Modelscited by 3
  15. 2021Advancing biological super-resolution microscopy through deep learning: a brief reviewcited by 2
  16. 2023Less is More -- Towards parsimonious multi-task models using structured sparsitycited by 2
  17. 2024Class-wise Generalization Error: an Information-Theoretic Analysiscited by 2
  18. 2025TITAN: Query-Token based Domain Adaptive Adversarial Learningcited by 2
  19. 2021PACMAN: PAC-style bounds accounting for the Mismatch between Accuracy and Negative log-losscited by 1
  20. 2021Spatial-Temporal-Fusion BNN: Variational Bayesian Feature Layercited by 1
  21. 2022Super-model ecosystem: A domain-adaptation perspectivecited by 1
  22. 2025Adaptive Classification of Interval-Valued Time Seriescited by 1
  23. 2025Calibrated and uncertain? Evaluating uncertainty estimates in binary classification modelscited by 1
  24. 2023Subspace-Configurable Networkscited by 0
  25. 2023XAI for In-hospital Mortality Prediction via Multimodal ICU Datacited by 0
  26. 2025Interval-Valued Time Series Classification Using DK-Distancecited by 0
  27. 2025SNAP: A Benchmark for Testing the Effects of Capture Conditions on Fundamental Vision Taskscited by 0
  28. 2025Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Powercited by 0
  29. 2026Understanding Deep Learning via Entropy Space Theorycited by 0