# The Modern Mathematics of Deep Learning

arXiv: https://arxiv.org/abs/2105.04026 · HTML: https://mlsanity.com/paper/2105.04026/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 May 2021.

## Built on by (most cited first, 30 of 63)

- [DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2403.03542/) (2024)
- [Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok](https://mlsanity.com/paper/2306.13253/) (2023)
- [Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionality](https://mlsanity.com/paper/2106.08900/) (2021)
- [Neural and spectral operator surrogates: unified construction and expression rate bounds](https://mlsanity.com/paper/2207.04950/) (2022)
- [Problem-Dependent Power of Quantum Neural Networks on Multi-Class Classification](https://mlsanity.com/paper/2301.01597/) (2022)
- [Training Fully Connected Neural Networks is existsℝ-Complete](https://mlsanity.com/paper/2204.01368/) (2022)
- [Robust SDE-Based Variational Formulations for Solving Linear PDEs via Deep Learning](https://mlsanity.com/paper/2206.10588/) (2022)
- [Dynamical systems' based neural networks](https://mlsanity.com/paper/2210.02373/) (2022)
- [Spiking neural networks for nonlinear regression](https://mlsanity.com/paper/2210.03515/) (2022)
- [Optimal learning of high-dimensional classification problems using deep neural networks](https://mlsanity.com/paper/2112.12555/) (2021)
- [Limitations of Deep Learning for Inverse Problems on Digital Hardware](https://mlsanity.com/paper/2202.13490/) (2022)
- [On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems](https://mlsanity.com/paper/2202.12262/) (2022)
- [To be or not to be stable, that is the question: understanding neural networks for inverse problems](https://mlsanity.com/paper/2211.13692/) (2022)
- [Approximation results for Gradient Descent trained Shallow Neural Networks in 1d](https://mlsanity.com/paper/2209.08399/) (2022)
- [ML Interpretability: Simple Isn't Easy](https://mlsanity.com/paper/2211.13617/) (2022)
- [Noise-Free Sampling Algorithms via Regularized Wasserstein Proximals](https://mlsanity.com/paper/2308.14945/) (2023)
- [Understanding the Difficulty of Solving Cauchy Problems with PINNs](https://mlsanity.com/paper/2405.02561/) (2024)
- [On generalization bounds for deep networks based on loss surface implicit regularization](https://mlsanity.com/paper/2201.04545/) (2022)
- [Decision boundaries and convex hulls in the feature space that deep learning functions learn from images](https://mlsanity.com/paper/2202.04052/) (2022)
- [Learning ReLU networks to high uniform accuracy is intractable](https://mlsanity.com/paper/2205.13531/) (2022)
- [Computability of Optimizers](https://mlsanity.com/paper/2301.06148/) (2023)
- [Error convergence and engineering-guided hyperparameter search of PINNs: towards optimized I-FENN performance](https://mlsanity.com/paper/2303.03918/) (2023)
- [Statistical learning theory and Occam's razor: The core argument](https://mlsanity.com/paper/2312.13842/) (2023)
- [The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations](https://mlsanity.com/paper/2401.13662/) (2024)
- [Stable Learning Using Spiking Neural Networks Equipped With Affine Encoders and Decoders](https://mlsanity.com/paper/2404.04549/) (2024)
- [SaDe: Learning Models that Provably Satisfy Domain Constraints](https://mlsanity.com/paper/2112.00552/) (2021)
- [Envisioning Future Deep Learning Theories: Some Basic Concepts and Characteristics](https://mlsanity.com/paper/2112.09741/) (2021)
- [Deep neural networks can stably solve high-dimensional, noisy, non-linear inverse problems](https://mlsanity.com/paper/2206.00934/) (2022)
- [On the Generalization and Approximation Capacities of Neural Controlled Differential Equations](https://mlsanity.com/paper/2305.16791/) (2023)
- [Component Transfer Learning for Deep RL Based on Abstract Representations](https://mlsanity.com/paper/2111.11525/) (2021)

## Abstract (arXiv)

We describe the new field of mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical framework of learning theory. These questions concern: the outstanding generalization power of overparametrized neural networks, the role of depth in deep architectures, the apparent absence of the curse of dimensionality, the surprisingly successful optimization performance despite the non-convexity of the problem, understanding what features are learned, why deep architectures perform exceptionally well in physical problems, and which fine aspects of an architecture affect the behavior of a learning task in which way. We present an overview of modern approaches that yield partial answers to these questions. For selected approaches, we describe the main ideas in more detail.