# Sparsity-depth Tradeoff in Infinitely Wide Deep Neural Networks

arXiv: https://arxiv.org/abs/2305.10550 · HTML: https://mlsanity.com/paper/2305.10550/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 13 of 13)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks](https://mlsanity.com/paper/2006.13198/) (2020)
- [Finite Versus Infinite Neural Networks: an Empirical Study](https://mlsanity.com/paper/2007.15801/) (2020)
- [Deep Equals Shallow for ReLU Networks in Kernel Regimes](https://mlsanity.com/paper/2009.14397/) (2020)
- [Exact posterior distributions of wide Bayesian neural networks](https://mlsanity.com/paper/2006.10541/) (2020)
- [Out-of-Distribution Generalization in Kernel Regression](https://mlsanity.com/paper/2106.02261/) (2021)
- [Uniform Generalization Bounds for Overparameterized Neural Networks](https://mlsanity.com/paper/2109.06099/) (2021)
- [Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm](https://mlsanity.com/paper/1905.10843/) (2019)
- [Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling](https://mlsanity.com/paper/2206.07673/) (2022)

## Abstract (arXiv)

We investigate how sparse neural activity affects the generalization performance of a deep Bayesian neural network at the large width limit. To this end, we derive a neural network Gaussian Process (NNGP) kernel with rectified linear unit (ReLU) activation and a predetermined fraction of active neurons. Using the NNGP kernel, we observe that the sparser networks outperform the non-sparse networks at shallow depths on a variety of datasets. We validate this observation by extending the existing theory on the generalization error of kernel-ridge regression.