Fast convergence rates of deep neural networks for classification
Not on the map. Why?
Abstract (arXiv)
We derive the fast convergence rates of a deep neural network (DNN) classifier with the rectified linear unit (ReLU) activation function learned using the hinge loss. We consider three cases for a true model: (1) a smooth decision boundary, (2) smooth conditional class probability, and (3) the margin condition (i.e., the probability of inputs near the decision boundary is small). We show that the DNN classifier learned using the hinge loss achieves fast rate convergences for all three cases provided that the architecture (i.e., the number of layers, number of nodes and sparsity). is carefully selected. An important implication is that DNN architectures are very flexible for use in various cases without much modification. In addition, we consider a DNN classifier learned by minimizing the cross-entropy, and show that the DNN classifier achieves a fast convergence rate under the condition that the conditional class probabilities of most data are sufficiently close to either 1 or zero. This assumption is not unusual for image recognition because human beings are extremely good at recognizing most images. To confirm our theoretical explanation, we present the results of a small numerical study conducted to compare the hinge loss and cross-entropy.
Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Dec 2018.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 7; 2022: 6; 2023: 11; 2024: 11; 2025: 6; 2026: 5.
46 papers.
Sort:
By area:
- 2021Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networkscited by 32
- 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
- 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
- 2021Convergence rates of deep ReLU networks for multiclass classificationcited by 21
- 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
- 2024Understanding the Learning Dynamics of Alignment with Human Feedbackcited by 15
- 2021Understanding Square Loss in Training Overparametrized Neural Network Classifierscited by 14
- 2022Active Learning with Neural Networks: Insights from Nonparametric Statisticscited by 13
- 2024On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networkscited by 13
- 2024On the rates of convergence for learning with convolutional neural networkscited by 10
- 2021Optimal learning of high-dimensional classification problems using deep neural networkscited by 9
- 2023Excess risk bound for deep learning under weak dependencecited by 9
34 more
- 2023Sparse-penalized deep neural networks estimator under weak dependencecited by 8
- 2024Misclassification bounds for PAC-Bayesian sparse deep learningcited by 7
- 2022Deep Neural Network Classifier for Multi-dimensional Functional Datacited by 5
- 2022Learning ReLU networks to high uniform accuracy is intractablecited by 5
- 2021Non-asymptotic Excess Risk Bounds for Classification with Deep Convolutional Neural Networkscited by 4
- 2022Analysis of convolutional neural network image classifiers in a rotationally symmetric modelcited by 4
- 2023Intrinsic and extrinsic deep learning on manifoldscited by 4
- 2023Classification of Data Generated by Gaussian Mixture Models Using Deep ReLU Networkscited by 4
- 2023On Excess Risk Convergence Rates of Neural Network Classifierscited by 3
- 2024Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimalitycited by 3
- 2026Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networkscited by 3
- 2021Analysis of convolutional neural network image classifiers in a hierarchical max-pooling model with additional local poolingcited by 2
- 2023Deep learning for psi-weakly dependent processescited by 2
- 2023Penalized deep neural networks estimator with general loss functions under weak dependencecited by 2
- 2023On the rate of convergence of an over-parametrized Transformer classifier learned by gradient descentcited by 2
- 2024High-dimensional classification problems with Barron regular boundaries under margin conditionscited by 2
- 2025Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionalitycited by 2
- 2025A general framework for deep learningcited by 2
- 2021VC dimension of partially quantized neural networks in the overparametrized regimecited by 1
- 2022Optimal Convergence Rates of Deep Convolutional Neural Networks: Additive Ridge Functionscited by 1
- 2023Multiclass classification for multidimensional functional data through deep neural networkscited by 1
- 2023Statistical learning by sparse deep neural networkscited by 1
- 2024Universal Consistency of Wide and Deep ReLU Neural Networks and Minimax Optimal Convergence Rates for Kolmogorov-Donoho Optimal Function Classescited by 1
- 2025Knowledge Distillation of Uncertainty using Deep Latent Factor Modelcited by 1
- 2022Orthogonal Series Estimation for the Ratio of Conditional Expectation Functionscited by 0
- 2024Dimension-independent learning rates for high-dimensional classification problemscited by 0
- 2024ReLU neural network approximation to piecewise constant functionscited by 0
- 2025Learning Guarantee of Reward Modeling Using Deep Neural Networkscited by 0
- 2025Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Conditioncited by 0
- 2025Minimax learning rates for estimating binary classifiers under margin conditionscited by 0
- 2026Model-agnostic information transfer and fusion for classification with label noisecited by 0
- 2026Rate-optimal neural boundary detection from unlabeled noisy imagescited by 0
- 2026Fast approximation and learning of binary classification tasks in o-minimal structures using ReLU neural networkscited by 0
- 2026Adaptive deep nonparametric regression from dependent data under covariate shiftcited by 0
- 2026Adaptive deep nonparametric regression from dependent data under covariate shift22 Jul 2026
- 2026Fast approximation and learning of binary classification tasks in o-minimal structures using ReLU neural networks29 Jun 2026
- 2026Rate-optimal neural boundary detection from unlabeled noisy images30 May 2026
- 2026Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks29 May 2026
- 2026Model-agnostic information transfer and fusion for classification with label noise28 Apr 2026
- 2025A general framework for deep learning29 Dec 2025
- 2025Knowledge Distillation of Uncertainty using Deep Latent Factor Model22 Oct 2025
- 2025Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality23 Jun 2025
- 2025Minimax learning rates for estimating binary classifiers under margin conditions15 May 2025
- 2025Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Condition13 May 2025
- 2025Learning Guarantee of Reward Modeling Using Deep Neural Networks10 May 2025
- 2024High-dimensional classification problems with Barron regular boundaries under margin conditions10 Dec 2024
34 more
- 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data11 Nov 2024
- 2024ReLU neural network approximation to piecewise constant functions21 Oct 2024
- 2024Dimension-independent learning rates for high-dimensional classification problems26 Sep 2024
- 2024On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks2 Sep 2024
- 2024Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality12 Jun 2024
- 2024Misclassification bounds for PAC-Bayesian sparse deep learning2 May 2024
- 2024Understanding the Learning Dynamics of Alignment with Human Feedback27 Mar 2024
- 2024On the rates of convergence for learning with convolutional neural networks25 Mar 2024
- 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models14 Jan 2024
- 2024Universal Consistency of Wide and Deep ReLU Neural Networks and Minimax Optimal Convergence Rates for Kolmogorov-Donoho Optimal Function Classes8 Jan 2024
- 2023On the rate of convergence of an over-parametrized Transformer classifier learned by gradient descent28 Dec 2023
- 2023Statistical learning by sparse deep neural networks15 Nov 2023
- 2023On Excess Risk Convergence Rates of Neural Network Classifiers26 Sep 2023
- 2023Classification of Data Generated by Gaussian Mixture Models Using Deep ReLU Networks15 Aug 2023
- 2023Multiclass classification for multidimensional functional data through deep neural networks22 May 2023
- 2023Penalized deep neural networks estimator with general loss functions under weak dependence10 May 2023
- 2023Sparse-penalized deep neural networks estimator under weak dependence2 Mar 2023
- 2023Intrinsic and extrinsic deep learning on manifolds16 Feb 2023
- 2023Excess risk bound for deep learning under weak dependence15 Feb 2023
- 2023Generalization Ability of Wide Neural Networks on ℝ12 Feb 2023
- 2023Deep learning for psi-weakly dependent processes1 Feb 2023
- 2022Orthogonal Series Estimation for the Ratio of Conditional Expectation Functions26 Dec 2022
- 2022Active Learning with Neural Networks: Insights from Nonparametric Statistics15 Oct 2022
- 2022Learning ReLU networks to high uniform accuracy is intractable26 May 2022
- 2022Deep Neural Network Classifier for Multi-dimensional Functional Data17 May 2022
- 2022Analysis of convolutional neural network image classifiers in a rotationally symmetric model11 May 2022
- 2022Optimal Convergence Rates of Deep Convolutional Neural Networks: Additive Ridge Functions24 Feb 2022
- 2021Optimal learning of high-dimensional classification problems using deep neural networks23 Dec 2021
- 2021Understanding Square Loss in Training Overparametrized Neural Network Classifiers7 Dec 2021
- 2021VC dimension of partially quantized neural networks in the overparametrized regime6 Oct 2021
- 2021Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks7 Sep 2021
- 2021Convergence rates of deep ReLU networks for multiclass classification2 Aug 2021
- 2021Analysis of convolutional neural network image classifiers in a hierarchical max-pooling model with additional local pooling31 May 2021
- 2021Non-asymptotic Excess Risk Bounds for Classification with Deep Convolutional Neural Networks1 May 2021
- 2021Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networkscited by 32
- 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
- 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
- 2021Convergence rates of deep ReLU networks for multiclass classificationcited by 21
- 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
- 2021Understanding Square Loss in Training Overparametrized Neural Network Classifierscited by 14
- 2022Active Learning with Neural Networks: Insights from Nonparametric Statisticscited by 13
- 2024On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networkscited by 13
- 2024On the rates of convergence for learning with convolutional neural networkscited by 10
- 2021Optimal learning of high-dimensional classification problems using deep neural networkscited by 9
- 2023Excess risk bound for deep learning under weak dependencecited by 9
- 2023Sparse-penalized deep neural networks estimator under weak dependencecited by 8
12 most cited of 44 All 44 (JSON)
- 2026Adaptive deep nonparametric regression from dependent data under covariate shift22 Jul 2026
- 2026Fast approximation and learning of binary classification tasks in o-minimal structures using ReLU neural networks29 Jun 2026
- 2026Rate-optimal neural boundary detection from unlabeled noisy images30 May 2026
- 2026Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks29 May 2026
- 2026Model-agnostic information transfer and fusion for classification with label noise28 Apr 2026
- 2025A general framework for deep learning29 Dec 2025
- 2025Knowledge Distillation of Uncertainty using Deep Latent Factor Model22 Oct 2025
- 2025Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality23 Jun 2025
- 2025Minimax learning rates for estimating binary classifiers under margin conditions15 May 2025
- 2025Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Condition13 May 2025
- 2024High-dimensional classification problems with Barron regular boundaries under margin conditions10 Dec 2024
- 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data11 Nov 2024
12 newest of 44 All 44 (JSON)