# Are wider nets better given the same number of parameters?

arXiv: https://arxiv.org/abs/2010.14495 · HTML: https://mlsanity.com/paper/2010.14495/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2020.

## Built on by (most cited first, 27 of 27)

- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [Stochastic Training is Not Necessary for Generalization](https://mlsanity.com/paper/2109.14119/) (2021)
- [What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory](https://mlsanity.com/paper/2105.03361/) (2021)
- [Why Random Pruning Is All We Need to Start Sparse](https://mlsanity.com/paper/2210.02412/) (2022)
- [DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability](https://mlsanity.com/paper/2210.00364/) (2022)
- [Deep Learning Meets Sparse Regularization: A Signal Processing Perspective](https://mlsanity.com/paper/2301.09554/) (2023)
- [The Role of Linear Layers in Nonlinear Interpolating Networks](https://mlsanity.com/paper/2202.00856/) (2022)
- [Sparse maximal update parameterization: A holistic approach to sparse training dynamics](https://mlsanity.com/paper/2405.15743/) (2024)
- [Masks, Signs, And Learning Rate Rewinding](https://mlsanity.com/paper/2402.19262/) (2024)
- [On Feature Learning in Neural Networks with Global Convergence Guarantees](https://mlsanity.com/paper/2204.10782/) (2022)
- [Laziness, Barren Plateau, and Noise in Machine Learning](https://mlsanity.com/paper/2206.09313/) (2022)
- [Finding Lottery Tickets in Vision Models via Data-driven Spectral Foresight Pruning](https://mlsanity.com/paper/2406.01820/) (2024)
- [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](https://mlsanity.com/paper/2311.11883/) (2023)
- [Understanding the effect of sparsity on neural networks robustness](https://mlsanity.com/paper/2206.10915/) (2022)
- [Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency](https://mlsanity.com/paper/2303.11525/) (2023)
- [ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Models](https://mlsanity.com/paper/2305.15598/) (2023)
- [Investigating the Impact of Model Width and Density on Generalization in Presence of Label Noise](https://mlsanity.com/paper/2208.08003/) (2022)
- [Understanding MLP-Mixer as a Wide and Sparse MLP](https://mlsanity.com/paper/2306.01470/) (2023)
- [Learning to Manipulate under Limited Information](https://mlsanity.com/paper/2401.16412/) (2024)
- [Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity](https://mlsanity.com/paper/2502.01330/) (2025)
- [Expand Neurons, Not Parameters](https://mlsanity.com/paper/2510.04500/) (2025)
- [REDS: Resource-Efficient Deep Subnetworks for Dynamic Resource Constraints](https://mlsanity.com/paper/2311.13349/) (2023)
- [Over-Parameterization and Generalization in Audio Classification](https://mlsanity.com/paper/2107.08933/) (2021)
- [MediSwift: Efficient Sparse Pre-trained Biomedical Language Models](https://mlsanity.com/paper/2403.00952/) (2024)
- [Matrix Completion via Nonsmooth Regularization of Fully Connected Neural Networks](https://mlsanity.com/paper/2403.10232/) (2024)
- [Sequential Reservoir Computing for Efficient High-Dimensional Spatiotemporal Forecasting](https://mlsanity.com/paper/2601.00172/) (2026)

## Abstract (arXiv)

Empirical studies demonstrate that the performance of neural networks improves with increasing number of parameters. In most of these studies, the number of parameters is increased by increasing the network width. This begs the question: Is the observed improvement due to the larger number of parameters, or is it due to the larger width itself? We compare different ways of increasing model width while keeping the number of parameters constant. We show that for models initialized with a random, static sparsity pattern in the weight tensors, network width is the determining factor for good performance, while the number of weights is secondary, as long as trainability is ensured. As a step towards understanding this effect, we analyze these models in the framework of Gaussian Process kernels. We find that the distance between the sparse finite-width model kernel and the infinite-width kernel at initialization is indicative of model performance.