Data to 7 Oct 2026

Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't

arXiv 2009.10713 ↗First posted 22 Sep 2020cs.LG cs.NA math.NAcited by 94

Not on the map. Why?

Abstract (arXiv)

The purpose of this article is to review the achievements made in the last few years towards the understanding of the reasons behind the success and subtleties of neural network-based machine learning. In the tradition of good old applied mathematics, we will not only give attention to rigorous mathematical results, but also the insight we have gained from careful numerical experiments as well as the analysis of simplified models. Along the way, we also list the open problems which we believe to be the most important topics for further study. This is not a complete overview over this quickly moving field, but we hope to provide a perspective which may be helpful especially to new researchers in the area.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 18; 2023: 16; 2024: 14; 2025: 9; 2026: 6.

94 papers.

Sort:

By area:

2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
2021Learning with invariances in random features and kernel modelscited by 89
2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
2021Noisy Recurrent Neural Networkscited by 42
2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 36
2021Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionalitycited by 34
2024Robust Concept Erasure Using Task Vectorscited by 27
2024Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modelingcited by 25
2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
2024Asymptotics of Random Feature Regression Beyond the Linear Scaling Regimecited by 22
2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 21
2021Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observationscited by 20

38 more

2022Infinite-width limit of deep linear neural networkscited by 20
2023Sampling weights of deep neural networkscited by 20
2021A Priori Generalization Error Analysis of Two-Layer Neural Networks for Solving High Dimensional Schr"odinger Eigenvalue Problemscited by 19
2021A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Equationscited by 18
2023Exploiting Noise as a Resource for Computation and Learning in Spiking Neural Networkscited by 17
2021Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanationscited by 15
2022Duality for Neural Networks through Reproducing Kernel Banach Spacescited by 15
2023A Brief Survey on the Approximation Theory for Sequence Modellingcited by 15
2024Convergence rates for the Adam optimizercited by 14
2021Exact Gap between Generalization Error and Uniform Convergence in Random Feature Modelscited by 13
2021An L2 Analysis of Reinforcement Learning in High Dimensions with Kernel and Neural Network Approximationcited by 13
2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
2022HARFE: Hard-Ridge Random Feature Expansioncited by 12
2023Universal approximation property of Banach space-valued random feature models including random neural networkscited by 12
2024Score-based generative models break the curse of dimensionality in learning a family of sub-Gaussian probability distributionscited by 12
2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 10
2022Asymptotic-Preserving Neural Networks for multiscale hyperbolic models of epidemic spreadcited by 10
2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
2021Stationary Density Estimation of Ito Diffusions Using Deep Learningcited by 9
2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
2021Optimal learning of high-dimensional classification problems using deep neural networkscited by 9
2021Generalization Error of GAN from the Discriminator's Perspectivecited by 8
2022To be or not to be stable, that is the question: understanding neural networks for inverse problemscited by 8
2021Cutting-edge 3D Medical Image Segmentation Methods in 2020: Are Happy Families All Alike?cited by 7
2021Bayesian Inference Forgettingcited by 7
2021Conditioning of Random Feature Matrices: Double Descent and Generalization Errorcited by 7
2022Understanding Deep Learning via Decision Boundarycited by 7
2022Approximation results for Gradient Descent trained Shallow Neural Networks in 1dcited by 7
2022A Mathematical Framework for Learning Probability Distributionscited by 7
2023Infinite-dimensional reservoir computingcited by 7
2024How Transformers Get Rich: Approximation and Dynamics Analysiscited by 7
2025A note on the physical interpretation of neural PDE'scited by 7
2025Sharp higher order convergence rates for the Adam optimizercited by 7
2022SRMD: Sparse Random Mode Decompositioncited by 6
2023On the existence of minimizers in shallow residual ReLU neural network optimization landscapescited by 6
2024Solving Partial Differential Equations with Random Feature Modelscited by 6

50 most cited of 94 All 94 (JSON)