Limitations of neural network training due to numerical instability of backpropagation
Not on the map. Why?
Abstract (arXiv)
We study the training of deep neural networks by gradient descent where floating-point arithmetic is used to compute the gradients. In this framework and under realistic assumptions, we demonstrate that it is highly unlikely to find ReLU neural networks that maintain, in the course of training with gradient descent, superlinearly many affine pieces with respect to their number of layers. In virtually all approximation theoretical arguments that yield high-order polynomial rates of approximation, sequences of ReLU neural networks with exponentially many affine pieces compared to their numbers of layers are used. As a consequence, we conclude that approximating sequences of ReLU neural networks resulting from gradient descent in practice differ substantially from theoretically constructed sequences. The assumptions and the theoretical results are compared to a numerical study, which yields concurring results.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 3 Oct 2022.
Builds on
28 of 50 references.
References · 28
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
- 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 581
- 2016Error bounds for approximations with deep ReLU networkscited by 567
- 2017Solving high-dimensional partial differential equations using deep learningcited by 549
- 2016Deep Convolutional Neural Network for Inverse Problems in Imagingcited by 431
- 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
- 2015Deep Learning with Limited Numerical Precisioncited by 368
- 2020Deep Learning Techniques for Inverse Problems in Imagingcited by 290
- 2020Model Reduction and Neural Networks for Parametric PDEscited by 257
16 more
- 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
- 2019Complexity of Linear Regions in Deep Networkscited by 166
- 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
- 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
- 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 90
- 2018NETT: Solving Inverse Problems with Deep Neural Networkscited by 89
- 2015Representation Benefits of Deep Feedforward Networkscited by 84
- 2019A Theoretical Analysis of Deep Neural Networks and Parametric PDEscited by 74
- 2015Provable approximation properties for deep neural networkscited by 60
- 2018A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equationscited by 55
- 2021Can stable and accurate neural networks be computed? -- On the barriers of deep learning and Smale's 18th problemcited by 53
- 2017Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networkscited by 21
- 2020Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularitiescited by 20
- 2022Limitations of Deep Learning for Inverse Problems on Digital Hardwarecited by 9
- 2022Deep neural networks can stably solve high-dimensional, noisy, non-linear inverse problemscited by 3
- 2018Exploration of Numerical Precision in Deep Neural Networkscited by 2
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.