# On the Depth of Deep Neural Networks: A Theoretical View

arXiv: https://arxiv.org/abs/1506.05232 · HTML: https://mlsanity.com/paper/1506.05232/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2015.

## Built on by (most cited first, 4 of 4)

- [Certified Robust Models with Slack Control and Large Lipschitz Constants](https://mlsanity.com/paper/2309.06166/) (2023)
- [The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs](https://mlsanity.com/paper/2302.06925/) (2023)
- [Pairwise Margin Maximization for Deep Neural Networks](https://mlsanity.com/paper/2110.04519/) (2021)
- [Quadratic Unconstrained Binary Optimisation for Training and Regularisation of Binary Neural Networks](https://mlsanity.com/paper/2601.00449/) (2026)

## Abstract (arXiv)

People believe that depth plays an important role in success of deep neural networks (DNN). However, this belief lacks solid theoretical justifications as far as we know. We investigate role of depth from perspective of margin bound. In margin bound, expected error is upper bounded by empirical margin error plus Rademacher Average (RA) based capacity term. First, we derive an upper bound for RA of DNN, and show that it increases with increasing depth. This indicates negative impact of depth on test performance. Second, we show that deeper networks tend to have larger representation power (measured by Betti numbers based complexity) than shallower networks in multi-class setting, and thus can lead to smaller empirical margin error. This implies positive impact of depth. The combination of these two results shows that for DNN with restricted number of hidden units, increasing depth is not always good since there is a tradeoff between positive and negative impacts. These results inspire us to seek alternative ways to achieve positive impact of depth, e.g., imposing margin-based penalty terms to cross entropy loss so as to reduce empirical margin error without increasing depth. Our experiments show that in this way, we achieve significantly better test performance.