# Classification with Deep Neural Networks and Logistic Loss

arXiv: https://arxiv.org/abs/2307.16792 · HTML: https://mlsanity.com/paper/2307.16792/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 55 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 7 of 7)

- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016)
- [Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks](https://mlsanity.com/paper/2006.07322/) (2020)
- [Deep Neural Networks Learn Non-Smooth Functions Effectively](https://mlsanity.com/paper/1802.04474/) (2018)
- [Early-stopped neural networks are consistent](https://mlsanity.com/paper/2106.05932/) (2021)
- [Understanding Square Loss in Training Overparametrized Neural Network Classifiers](https://mlsanity.com/paper/2112.03657/) (2021)
- [Non-asymptotic Excess Risk Bounds for Classification with Deep Convolutional Neural Networks](https://mlsanity.com/paper/2105.00292/) (2021)

## Built on by (most cited first, 16 of 16)

- [Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss](https://mlsanity.com/paper/2402.00152/) (2024)
- [On the rates of convergence for learning with convolutional neural networks](https://mlsanity.com/paper/2403.16459/) (2024)
- [Robust deep learning from weakly dependent data](https://mlsanity.com/paper/2405.05081/) (2024)
- [Generalization analysis with deep ReLU networks for metric and similarity learning](https://mlsanity.com/paper/2405.06415/) (2024)
- [A direct proof of a unified law of robustness for Bregman divergence losses](https://mlsanity.com/paper/2405.16639/) (2024)
- [Minimax optimality of deep neural networks on dependent data via PAC-Bayes bounds](https://mlsanity.com/paper/2410.21702/) (2024)
- [Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality](https://mlsanity.com/paper/2406.08321/) (2024)
- [Deep learning with missing data](https://mlsanity.com/paper/2504.15388/) (2025)
- [A general framework for deep learning](https://mlsanity.com/paper/2512.23425/) (2025)
- [Universal Consistency of Wide and Deep ReLU Neural Networks and Minimax Optimal Convergence Rates for Kolmogorov-Donoho Optimal Function Classes](https://mlsanity.com/paper/2401.04286/) (2024)
- [Component-based Sketching for Deep ReLU Nets](https://mlsanity.com/paper/2409.14174/) (2024)
- [Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Condition](https://mlsanity.com/paper/2505.08262/) (2025)
- [Deep regression learning from dependent observations with minimum error entropy principle](https://mlsanity.com/paper/2603.11138/) (2026)
- [PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory](https://mlsanity.com/paper/2604.17219/) (2026)
- [Ratio-based Loss Functions](https://mlsanity.com/paper/2605.05808/) (2026)
- [Rate-optimal neural boundary detection from unlabeled noisy images](https://mlsanity.com/paper/2606.00715/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) trained with the logistic loss (i.e., the cross entropy loss) have made impressive advancements in various binary classification tasks. However, generalization analysis for binary classification with DNNs and logistic loss remains scarce. The unboundedness of the target function for the logistic loss is the main obstacle to deriving satisfactory generalization bounds. In this paper, we aim to fill this gap by establishing a novel and elegant oracle-type inequality, which enables us to deal with the boundedness restriction of the target function, and using it to derive sharp convergence rates for fully connected ReLU DNN classifiers trained with logistic loss. In particular, we obtain optimal convergence rates (up to log factors) only requiring the H"older smoothness of the conditional class probability η of data. Moreover, we consider a compositional assumption that requires η to be the composition of several vector-valued functions of which each component function is either a maximum value function or a H"older smooth function only depending on a small number of its input variables. Under this assumption, we derive optimal convergence rates (up to log factors) which are independent of the input dimension of data. This result explains why DNN classifiers can perform well in practical high-dimensional classification problems. Besides the novel oracle-type inequality, the sharp convergence rates given in our paper also owe to a tight error bound for approximating the natural logarithm function near zero (where it is unbounded) by ReLU DNNs. In addition, we justify our claims for the optimality of rates by proving corresponding minimax lower bounds. All these results are new in the literature and will deepen our theoretical understanding of classification with DNNs.