mlsanityData to 14 Oct 2026

Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Loss

arXiv 2309.06774 ↗First posted 13 Sep 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Although deep learning (DL) has led to several breakthroughs in many disciplines, the fundamental understanding on why and how DL is empirically successful remains elusive. To attack this fundamental problem and unravel the mysteries behind DL's empirical successes, significant innovations toward a unified theory of DL have been made. Although these innovations encompass nearly fundamental advances in optimization, generalization, and approximation, no work has quantified the testing performance of a DL-based algorithm employed to solve a pattern classification problem. To overcome this fundamental challenge in part, this paper exposes the fundamental testing performance limits of DL-based binary classifiers trained with hinge loss. For binary classifiers that are based on deep rectified linear unit (ReLU) feedforward neural networks (FNNs) and deep FNNs with ReLU and Tanh activation, we derive their respective novel asymptotic testing performance limits, which are validated by extensive computer experiments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 66 of its 124 references that are in the map. First posted on arXiv 13 Sep 2023.

Builds on

66 of 124 references.

References · 66

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  8. 2018On Lazy Training in Differentiable Programmingcited by 675
  9. 2017Solving high-dimensional partial differential equations using deep learningcited by 556
  10. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  11. 2019Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processingcited by 421
  12. 2018Deep Learning for Generic Object Detection: A Surveycited by 392
38 more
  1. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  2. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  3. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 263
  4. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  5. 2021Deep learning: a statistical viewpointcited by 240
  6. 2017An Introduction to Deep Learning for the Physical Layercited by 233
  7. 2018Deep Learning: A Critical Appraisalcited by 217
  8. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  9. 2015The Power of Depth for Feedforward Neural Networkscited by 183
  10. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 179
  11. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  12. 2018Applications of Deep Reinforcement Learning in Communications and Networking: A Surveycited by 167
  13. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  14. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  15. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  16. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  17. 2018Deep Learning in Mobile and Wireless Networking: A Surveycited by 118
  18. 2022Causal Machine Learning: A Survey and Open Problemscited by 110
  19. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
  20. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  21. 2016Why does deep and cheap learning work so well?cited by 100
  22. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  23. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 95
  24. 2019Deep Neural Network Approximation Theorycited by 93
  25. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 91
  26. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  27. 2018ResNet with one-neuron hidden layers is a Universal Approximatorcited by 79
  28. 2016Towards an integration of deep learning and neurosciencecited by 78
  29. 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
  30. 2014On the Computational Efficiency of Training Neural Networkscited by 71
  31. 2019Optimization for deep learning: theory and algorithmscited by 70
  32. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  33. 2016Deep vs. shallow networks : An approximation theory perspectivecited by 63
  34. 2021The Modern Mathematics of Deep Learningcited by 63
  35. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  36. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
  37. 2020The gap between theory and practice in function approximation with deep neural networkscited by 54
  38. 2018Quantum Entanglement in Deep Learning Architecturescited by 50

50 most cited of 66 All 66 (JSON)

Built on by

Not cited yet.