# A Deep Conditioning Treatment of Neural Networks

arXiv: https://arxiv.org/abs/2002.01523 · HTML: https://mlsanity.com/paper/2002.01523/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Feb 2020.

## Built on by (most cited first, 15 of 15)

- [Batch Normalization Orthogonalizes Representations in Deep Random Networks](https://mlsanity.com/paper/2106.03970/) (2021)
- [Hardness of Noise-Free Learning for Two-Hidden-Layer Neural Networks](https://mlsanity.com/paper/2202.05258/) (2022)
- [From Local Pseudorandom Generators to Hardness of Learning](https://mlsanity.com/paper/2101.08303/) (2021)
- [AESPA: Accuracy Preserving Low-degree Polynomial Activation for Fast Private Inference](https://mlsanity.com/paper/2201.06699/) (2022)
- [Learning (Very) Simple Generative Models Is Hard](https://mlsanity.com/paper/2205.16003/) (2022)
- [On the average-case complexity of learning output distributions of quantum circuits](https://mlsanity.com/paper/2305.05765/) (2023)
- [Cutting the Skip: Training Residual-Free Transformers](https://mlsanity.com/paper/2510.00345/) (2025)
- [A quantum algorithm for training wide and deep classical neural networks](https://mlsanity.com/paper/2107.09200/) (2021)
- [Leaner Transformers: More Heads, Less Depth](https://mlsanity.com/paper/2505.20802/) (2025)
- [Universal scaling laws in the gradient descent training of neural networks](https://mlsanity.com/paper/2105.00507/) (2021)
- [Feel-Good Thompson Sampling for Contextual Bandits: a Markov Chain Monte Carlo Showdown](https://mlsanity.com/paper/2507.15290/) (2025)
- [Enhancing Transformers Through Conditioned Embedded Tokens](https://mlsanity.com/paper/2505.12789/) (2025)
- [Spectral Conditioning of Attention Improves Transformer Performance](https://mlsanity.com/paper/2603.07162/) (2026)
- [Most Neural Networks Are Almost Learnable](https://mlsanity.com/paper/2305.16508/) (2023)
- [Conditioned Initialization for Attention](https://mlsanity.com/paper/2609.07086/) (2026)

## Abstract (arXiv)

We study the role of depth in training randomly initialized overparameterized neural networks. We give a general result showing that depth improves trainability of neural networks by improving the conditioning of certain kernel matrices of the input data. This result holds for arbitrary non-linear activation functions under a certain normalization. We provide versions of the result that hold for training just the top layer of the neural network, as well as for training all layers, via the neural tangent kernel. As applications of these general results, we provide a generalization of the results of Das et al. (2019) showing that learnability of deep random neural networks with a large class of non-linear activations degrades exponentially with depth. We also show how benign overfitting can occur in deep neural networks via the results of Bartlett et al. (2019b). We also give experimental evidence that normalized versions of ReLU are a viable alternative to more complex operations like Batch Normalization in training deep neural networks.