mlsanityData to 7 Oct 2026

Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Condition

arXiv 2505.08262 ↗First posted 13 May 2025cs.LG math.ST stat.THcited by 0

Not on the map. Why?

Abstract (arXiv)

We study the classical binary classification problem for hypothesis spaces of Deep Neural Networks (DNNs) under Tsybakov's low-noise condition with exponent q>0, as well as its limit case q=∞, which we refer to as the hard margin condition. We demonstrate that, for a wide range of commonly used activation functions (including but not limited to ReLU, LeakyReLU, ELU, CELU, SELU, Softplus, GELU, SiLU, Swish, Mish, and Softmax), DNN solutions to the empirical risk minimization (ERM) problem with square loss surrogate and ℓp penalty on the weights (0<p<∞) can achieve excess risk bounds of order Oleft(n-αright) for α close to 1 under the low-noise condition, and for arbitrarily large α>1 under the hard-margin condition, provided that the Bayes regression function η satisfies a distribution-adapted smoothness condition relative to the marginal data distribution ρX. Furthermore, when the activation function is chosen as tanh or sigmoid, we show that the same rates follow from the standard assumption that η∈ Cs. Finally, we establish minimax lower bounds, showing that these rates cannot be improved upon whenever q≥2. Our proof relies on a novel decomposition of the excess risk for general ERM-based classifiers which might be of independent interest.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 71 references that are in the map. First posted on arXiv 13 May 2025.

Builds on

38 of 71 references.

References · 38

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  4. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  5. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  6. 2016Error bounds for approximations with deep ReLU networkscited by 567
  7. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  8. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  9. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  10. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  11. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  12. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
26 more
  1. 2018Universality of Deep Convolutional Neural Networkscited by 176
  2. 2020Deep Network Approximation for Smooth Functionscited by 163
  3. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  4. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  5. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  6. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  7. 2021On the approximation of functions by tanh neural networkscited by 81
  8. 2016Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networkscited by 58
  9. 2018Fast convergence rates of deep neural networks for classificationcited by 46
  10. 2018Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equationscited by 44
  11. 2018Global Convergence of Block Coordinate Descent in Deep Learningcited by 38
  12. 2016Calculus of the exponent of Kurdyka-Lojasiewicz inequality and its applications to linear convergence of first-order methodscited by 35
  13. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  14. 2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
  15. 2022Improved Bounds on Neural Complexity for Representing Piecewise Linear Functionscited by 26
  16. 2023Nonparametric regression using over-parameterized shallow ReLU neural networkscited by 24
  17. 2021Convergence rates of deep ReLU networks for multiclass classificationcited by 21
  18. 2023Classification with Deep Neural Networks and Logistic Losscited by 16
  19. 2021Understanding Square Loss in Training Overparametrized Neural Network Classifierscited by 14
  20. 2021Optimal learning of high-dimensional classification problems using deep neural networkscited by 9
  21. 2024Don't Fear Peculiar Activation Functions: EUAF and Beyondcited by 7
  22. 2022Multiclass learning with margin: exponential rates with no bias-variance trade-offcited by 5
  23. 2022Convergence Rates of Training Deep Neural Networks via Alternating Minimization Methodscited by 4
  24. 2019Approximation in Lp(μ) with deep ReLU neural networkscited by 3
  25. 2022A Case of Exponential Convergence Rates for SVMcited by 3
  26. 2023On Excess Risk Convergence Rates of Neural Network Classifierscited by 3

Built on by

Not cited yet.