# Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

arXiv: https://arxiv.org/abs/2310.20360 · HTML: https://mlsanity.com/paper/2310.20360/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2023.

## Built on by (most cited first, 20 of 20)

- [Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates](https://mlsanity.com/paper/2407.08100/) (2024)
- [Convergence rates for the Adam optimizer](https://mlsanity.com/paper/2407.21078/) (2024)
- [ODE approximation for the Adam algorithm: General and overparametrized setting](https://mlsanity.com/paper/2511.04622/) (2025)
- [Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks](https://mlsanity.com/paper/2402.05155/) (2024)
- [An Overview on Machine Learning Methods for Partial Differential Equations: from Physics Informed Neural Networks to Deep Operator Learning](https://mlsanity.com/paper/2408.13222/) (2024)
- [Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networks](https://mlsanity.com/paper/2503.01660/) (2025)
- [Towards Lossless Implicit Neural Representation via Bit Plane Decomposition](https://mlsanity.com/paper/2502.21001/) (2025)
- [Multilevel Picard approximations and deep neural networks with ReLU, leaky ReLU, and softplus activation overcome the curse of dimensionality when approximating semilinear parabolic partial differential equations in Lp-sense](https://mlsanity.com/paper/2409.20431/) (2024)
- [Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method](https://mlsanity.com/paper/2603.18899/) (2026)
- [Reconciling Privacy and Explainability in High-Stakes: A Systematic Inquiry](https://mlsanity.com/paper/2412.20798/) (2024)
- [PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning](https://mlsanity.com/paper/2505.22085/) (2025)
- [Central limit theorem for the averaged Adam optimizer](https://mlsanity.com/paper/2606.21433/) (2026)
- [Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks](https://mlsanity.com/paper/2607.04233/) (2026)
- [Towards an Algebraic Framework For Approximating Functions Using Neural Network Polynomials](https://mlsanity.com/paper/2402.01058/) (2024)
- [An overview of diffusion models for generative artificial intelligence](https://mlsanity.com/paper/2412.01371/) (2024)
- [High-Order Tensor Regression in Sparse Convolutional Neural Networks](https://mlsanity.com/paper/2501.01239/) (2025)
- [Interval-Valued Time Series Classification Using DK-Distance](https://mlsanity.com/paper/2504.04667/) (2025)
- [Error analysis for the deep Kolmogorov method](https://mlsanity.com/paper/2508.17167/) (2025)
- [Walk-on-Spheres Monte Carlo and deep neural network approximations of elliptic PDEs with drift and killing](https://mlsanity.com/paper/2608.09494/) (2026)
- [Convergence rates for the RMSprop optimizer with full control of the hyperparameters](https://mlsanity.com/paper/2608.30382/) (2026)

## Abstract (arXiv)

This book aims to provide an introduction to the topic of deep learning algorithms. We review essential components of deep learning algorithms in full mathematical detail including different artificial neural network (ANN) architectures (such as fully-connected feedforward ANNs, convolutional ANNs, recurrent ANNs, residual ANNs, and ANNs with batch normalization) and different optimization algorithms (such as the basic stochastic gradient descent (SGD) method, accelerated methods, and adaptive methods). We also cover several theoretical aspects of deep learning algorithms such as approximation capacities of ANNs (including a calculus for ANNs), optimization theory (including Kurdyka-Lojasiewicz inequalities), and generalization errors. In the last part of the book some deep learning approximation methods for PDEs are reviewed including physics-informed neural networks (PINNs) and deep Galerkin methods. We hope that this book will be useful for students and scientists who do not yet have any background in deep learning at all and would like to gain a solid foundation as well as for practitioners who would like to obtain a firmer mathematical understanding of the objects and methods considered in deep learning.