Transgressing the boundaries: towards a rigorous understanding of deep learning and its (non-)robustness
Not on the map. Why?
Abstract (arXiv)
The recent advances in machine learning in various fields of applications can be largely attributed to the rise of deep learning (DL) methods and architectures. Despite being a key technology behind autonomous cars, image processing, speech recognition, etc., a notorious problem remains the lack of theoretical understanding of DL and related interpretability and (adversarial) robustness issues. Understanding the specifics of DL, as compared to, say, other forms of nonlinear regression methods or statistical learning, is interesting from a mathematical perspective, but at the same time it is of crucial importance in practice: treating neural networks as mere black boxes might be sufficient in certain cases, but many applications require waterproof performance guarantees and a deeper understanding of what could go wrong and why it could go wrong. It is probably fair to say that, despite being mathematically well founded as a method to approximate complicated functions, DL is mostly still more like modern alchemy that is firmly in the hands of engineers and computer scientists. Nevertheless, it is evident that certain specifics of DL that could explain its success in applications demands systematic mathematical approaches. In this work, we review robustness issues of DL and particularly bridge concerns and attempts from approximation theory to statistical learning theory. Further, we review Bayesian Deep Learning as a means for uncertainty quantification and rigorous explainability.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 50 of its 98 references that are in the map. First posted on arXiv 5 Jul 2023.
Builds on
50 of 98 references.
References · 50
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2013Intriguing properties of neural networkscited by 5,114
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
- 2016Adversarial examples in the physical worldcited by 1,762
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
- 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
38 more
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
- 2019Benign Overfitting in Linear Regressioncited by 544
- 2017Adversarial Patchcited by 514
- 2020Overfitting in adversarially robust deep learningcited by 504
- 2017Sharp Minima Can Generalize For Deep Netscited by 462
- 2019On Evaluating Adversarial Robustnesscited by 442
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
- 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
- 2017Detecting Adversarial Samples from Artifactscited by 244
- 2018Techniques for Interpretable Machine Learningcited by 237
- 2018Generalisation in humans and deep neural networkscited by 237
- 2017Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equationscited by 188
- 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
- 2020Implicit Gradient Regularizationcited by 145
- 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
- 2021On the Convergence and Robustness of Adversarial Trainingcited by 127
- 2019Deep neural network solution of the electronic Schr"odinger equationcited by 124
- 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
- 2020Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path spacecited by 80
- 2020VarGrad: A Low-Variance Gradient Estimator for Variational Inferencecited by 73
- 2018Adv-BNN: Improved Adversarial Defense through Robust Bayesian Neural Networkcited by 68
- 2021The Modern Mathematics of Deep Learningcited by 63
- 2018Multi-Temporal Land Cover Classification with Sequential Recurrent Encoderscited by 61
- 2019Quality of Uncertainty Quantification for Bayesian Neural Network Inferencecited by 58
- 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 58
- 2020Robustness of Bayesian Neural Networks to Gradient-Based Attackscited by 48
- 2020Multiple Descent: Design Your Own Generalization Curvecited by 44
- 2018Learning The Invisible: A Hybrid Deep Learning-Shearlet Framework for Limited Angle Computed Tomographycited by 29
- 2021Neural networks-based algorithms for stochastic control and PDEs in financecited by 26
- 2022The no-free-lunch theorems of supervised learningcited by 16
- 2019How degenerate is the parametrization of neural networks with the ReLU activation function?cited by 15
- 2020AMAGOLD: Amortized Metropolis Adjustment for Efficient Stochastic Gradient MCMCcited by 13
- 2019Comment on "Adv-BNN: Improved Adversarial Defense through Robust Bayesian Neural Network"cited by 12
- 2017Adversarial Phenomenon in the Eyes of Bayesian Deep Learningcited by 11
- 2021SGD Implicitly Regularizes Generalization Errorcited by 10
- 2018Sufficient Conditions for Idealised Models to Have No Adversarial Examples: a Theoretical and Empirical Study with Bayesian Neural Networkscited by 9
- 2020Generalization and Memorization: The Bias Potential Modelcited by 9
- 2020Free Energy Minimization: A Unified Framework for Modelling, Inference, Learning,and Optimizationcited by 4
Built on by
Citing papers per year: 2024: 1.
1 paper.