# Fast convergence rates of deep neural networks for classification

arXiv: https://arxiv.org/abs/1812.03599 · HTML: https://mlsanity.com/paper/1812.03599/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2018.

## Built on by (most cited first, 30 of 46)

- [Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks](https://mlsanity.com/paper/2109.02832/) (2021)
- [Generalization Ability of Wide Neural Networks on ℝ](https://mlsanity.com/paper/2302.05933/) (2023)
- [Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data](https://mlsanity.com/paper/2411.06646/) (2024)
- [Convergence rates of deep ReLU networks for multiclass classification](https://mlsanity.com/paper/2108.00969/) (2021)
- [A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models](https://mlsanity.com/paper/2401.07187/) (2024)
- [Understanding the Learning Dynamics of Alignment with Human Feedback](https://mlsanity.com/paper/2403.18742/) (2024)
- [Understanding Square Loss in Training Overparametrized Neural Network Classifiers](https://mlsanity.com/paper/2112.03657/) (2021)
- [Active Learning with Neural Networks: Insights from Nonparametric Statistics](https://mlsanity.com/paper/2210.08367/) (2022)
- [On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks](https://mlsanity.com/paper/2409.00901/) (2024)
- [On the rates of convergence for learning with convolutional neural networks](https://mlsanity.com/paper/2403.16459/) (2024)
- [Optimal learning of high-dimensional classification problems using deep neural networks](https://mlsanity.com/paper/2112.12555/) (2021)
- [Excess risk bound for deep learning under weak dependence](https://mlsanity.com/paper/2302.07503/) (2023)
- [Sparse-penalized deep neural networks estimator under weak dependence](https://mlsanity.com/paper/2303.01406/) (2023)
- [Misclassification bounds for PAC-Bayesian sparse deep learning](https://mlsanity.com/paper/2405.01304/) (2024)
- [Deep Neural Network Classifier for Multi-dimensional Functional Data](https://mlsanity.com/paper/2205.08592/) (2022)
- [Learning ReLU networks to high uniform accuracy is intractable](https://mlsanity.com/paper/2205.13531/) (2022)
- [Non-asymptotic Excess Risk Bounds for Classification with Deep Convolutional Neural Networks](https://mlsanity.com/paper/2105.00292/) (2021)
- [Analysis of convolutional neural network image classifiers in a rotationally symmetric model](https://mlsanity.com/paper/2205.05500/) (2022)
- [Intrinsic and extrinsic deep learning on manifolds](https://mlsanity.com/paper/2302.08606/) (2023)
- [Classification of Data Generated by Gaussian Mixture Models Using Deep ReLU Networks](https://mlsanity.com/paper/2308.08030/) (2023)
- [On Excess Risk Convergence Rates of Neural Network Classifiers](https://mlsanity.com/paper/2309.15075/) (2023)
- [Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality](https://mlsanity.com/paper/2406.08321/) (2024)
- [Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks](https://mlsanity.com/paper/2605.31152/) (2026)
- [Analysis of convolutional neural network image classifiers in a hierarchical max-pooling model with additional local pooling](https://mlsanity.com/paper/2106.05233/) (2021)
- [Deep learning for psi-weakly dependent processes](https://mlsanity.com/paper/2302.00333/) (2023)
- [Penalized deep neural networks estimator with general loss functions under weak dependence](https://mlsanity.com/paper/2305.06230/) (2023)
- [On the rate of convergence of an over-parametrized Transformer classifier learned by gradient descent](https://mlsanity.com/paper/2312.17007/) (2023)
- [High-dimensional classification problems with Barron regular boundaries under margin conditions](https://mlsanity.com/paper/2412.07312/) (2024)
- [Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality](https://mlsanity.com/paper/2506.19144/) (2025)
- [A general framework for deep learning](https://mlsanity.com/paper/2512.23425/) (2025)

## Abstract (arXiv)

We derive the fast convergence rates of a deep neural network (DNN) classifier with the rectified linear unit (ReLU) activation function learned using the hinge loss. We consider three cases for a true model: (1) a smooth decision boundary, (2) smooth conditional class probability, and (3) the margin condition (i.e., the probability of inputs near the decision boundary is small). We show that the DNN classifier learned using the hinge loss achieves fast rate convergences for all three cases provided that the architecture (i.e., the number of layers, number of nodes and sparsity). is carefully selected. An important implication is that DNN architectures are very flexible for use in various cases without much modification. In addition, we consider a DNN classifier learned by minimizing the cross-entropy, and show that the DNN classifier achieves a fast convergence rate under the condition that the conditional class probabilities of most data are sufficiently close to either 1 or zero. This assumption is not unusual for image recognition because human beings are extremely good at recognizing most images. To confirm our theoretical explanation, we present the results of a small numerical study conducted to compare the hinge loss and cross-entropy.