mlsanityData to 14 Oct 2026

Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks

arXiv 2111.12963 ↗First posted 25 Nov 2021cs.LG stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

Among the several paradigms of artificial intelligence (AI) or machine learning (ML), a remarkably successful paradigm is deep learning. Deep learning's phenomenal success has been hoped to be interpreted via fundamental research on the theory of deep learning. Accordingly, applied research on deep learning has spurred the theory of deep learning-oriented depth and breadth of developments. Inspired by such developments, we pose these fundamental questions: can we accurately approximate an arbitrary matrix-vector product using deep rectified linear unit (ReLU) feedforward neural networks (FNNs)? If so, can we bound the resulting approximation error? In light of these questions, we derive error bounds in Lebesgue and Sobolev norms that comprise our developed deep approximation theory. Guided by this theory, we have successfully trained deep ReLU FNNs whose test results justify our developed theory. The developed theory is also applicable for guiding and easing the training of teacher deep ReLU FNNs in view of the emerging teacher-student AI or ML paradigms that are essential for solving several AI or ML problems in wireless communications and signal processing; network science and graph signal processing; and network neuroscience and brain physics.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 80 of its 135 references that are in the map. First posted on arXiv 25 Nov 2021.

Builds on

80 of 135 references.

References · 80

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  9. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  10. 2017Generative Adversarial Networks: An Overviewcited by 590
  11. 2016Error bounds for approximations with deep ReLU networkscited by 574
  12. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
38 more
  1. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  2. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  3. 2017Graph Signal Processing: Overview, Challenges and Applicationscited by 413
  4. 2018Deep Learning for Generic Object Detection: A Surveycited by 392
  5. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 368
  6. 2015Recent Advances in Convolutional Neural Networkscited by 360
  7. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  8. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  9. 2015LSTM: A Search Space Odysseycited by 316
  10. 2018Model compression via distillation and quantizationcited by 289
  11. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  12. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
  13. 2016Deep Learning without Poor Local Minimacited by 234
  14. 2015Unifying distillation and privileged informationcited by 233
  15. 2016Deep Information Propagationcited by 195
  16. 2018To understand deep learning we need to understand kernel learningcited by 187
  17. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  18. 2015The Power of Depth for Feedforward Neural Networkscited by 183
  19. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 179
  20. 2018Applications of Deep Reinforcement Learning in Communications and Networking: A Surveycited by 167
  21. 2017Learning from Noisy Labels with Distillationcited by 161
  22. 2021Towards Understanding Knowledge Distillationcited by 158
  23. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  24. 2017Generalization in Deep Learningcited by 146
  25. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  26. 2018Recent Advances in Autoencoder-Based Representation Learningcited by 133
  27. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
  28. 2016Why does deep and cheap learning work so well?cited by 100
  29. 2020Convolutional Networks with Dense Connectivitycited by 98
  30. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
  31. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  32. 2019Deep Neural Network Approximation Theorycited by 93
  33. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 91
  34. 2015Representation Benefits of Deep Feedforward Networkscited by 86
  35. 2018ResNet with one-neuron hidden layers is a Universal Approximatorcited by 79
  36. 2016Towards an integration of deep learning and neurosciencecited by 78
  37. 2019A Theoretical Analysis of Deep Neural Networks and Parametric PDEscited by 74
  38. 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1.

2 papers.

  1. 2022Blind Estimation of a Doubly Selective OFDM Channel: A Deep Learning Algorithm and Theorycited by 2
  2. 2023Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Losscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Three Quantization Regimes for ReLU Networks
  2. 2022Approximation speed of quantized vs. unquantized ReLU neural networks and beyond
  3. 2023On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem
  4. 2024A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizations
  5. 2025Principled Approximation Methods for Efficient and Scalable Deep Learning
  6. 2024Covering Numbers for Deep ReLU Networks with Applications to Function Approximation and Nonparametric Regression
  7. 2024Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts
  8. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting