# Depth Selection for Deep ReLU Nets in Feature Extraction and Generalization

arXiv: https://arxiv.org/abs/2004.00245 · HTML: https://mlsanity.com/paper/2004.00245/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2020.

## Built on by (most cited first, 13 of 13)

- [A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models](https://mlsanity.com/paper/2401.07187/) (2024)
- [Rates of Approximation by ReLU Shallow Neural Networks](https://mlsanity.com/paper/2307.12461/) (2023)
- [Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets](https://mlsanity.com/paper/2111.14039/) (2021)
- [Deep Convolutional Neural Networks with Zero-Padding: Feature Extraction and Learning](https://mlsanity.com/paper/2307.16203/) (2023)
- [Learning Sparse Compositional Functions with Norm-Constrained Neural Networks](https://mlsanity.com/paper/2605.25608/) (2026)
- [Fine-grained Analysis of Non-parametric Estimation for Pairwise Learning](https://mlsanity.com/paper/2305.19640/) (2023)
- [Convergence Analysis for Deep Sparse Coding via Convolutional Neural Networks](https://mlsanity.com/paper/2408.05540/) (2024)
- [Optimal Approximation and Learning Rates for Deep Convolutional Neural Networks](https://mlsanity.com/paper/2308.03259/) (2023)
- [Can overfitted deep neural networks in adversarial training generalize? -- An approximation viewpoint](https://mlsanity.com/paper/2401.13624/) (2024)
- [Component-based Sketching for Deep ReLU Nets](https://mlsanity.com/paper/2409.14174/) (2024)
- [Lifting the Veil: Unlocking the Power of Depth in Q-learning](https://mlsanity.com/paper/2310.17915/) (2023)
- [Feature Qualification by Deep Nets: A Constructive Approach](https://mlsanity.com/paper/2503.18676/) (2025)
- [Centroid Decision Forest](https://mlsanity.com/paper/2503.19306/) (2025)

## Abstract (arXiv)

Deep learning is recognized to be capable of discovering deep features for representation learning and pattern recognition without requiring elegant feature engineering techniques by taking advantage of human ingenuity and prior knowledge. Thus it has triggered enormous research activities in machine learning and pattern recognition. One of the most important challenge of deep learning is to figure out relations between a feature and the depth of deep neural networks (deep nets for short) to reflect the necessity of depth. Our purpose is to quantify this feature-depth correspondence in feature extraction and generalization. We present the adaptivity of features to depths and vice-verse via showing a depth-parameter trade-off in extracting both single feature and composite features. Based on these results, we prove that implementing the classical empirical risk minimization on deep nets can achieve the optimal generalization performance for numerous learning tasks. Our theoretical results are verified by a series of numerical experiments including toy simulations and a real application of earthquake seismic intensity prediction.