# Big Neural Networks Waste Capacity

arXiv: https://arxiv.org/abs/1301.3583 · HTML: https://mlsanity.com/paper/1301.3583/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2013.

## Built on by (most cited first, 10 of 10)

- [Probing the Purview of Neural Networks via Gradient Analysis](https://mlsanity.com/paper/2304.02834/) (2023)
- [FRL: Federated Rank Learning](https://mlsanity.com/paper/2110.04350/) (2021)
- [XnODR and XnIDR: Two Accurate and Fast Fully Connected Layers For Convolutional Neural Networks](https://mlsanity.com/paper/2111.10854/) (2021)
- [Not All Edges are Equally Robust: Evaluating the Robustness of Ranking-Based Federated Learning](https://mlsanity.com/paper/2503.08976/) (2025)
- [E2FL: Equal and Equitable Federated Learning](https://mlsanity.com/paper/2205.10454/) (2022)
- [Neural Architecture Search: Two Constant Shared Weights Initialisations](https://mlsanity.com/paper/2302.04406/) (2023)
- [Enhanced Multi-Class Classification of Gastrointestinal Endoscopic Images with Interpretable Deep Learning Model](https://mlsanity.com/paper/2503.00780/) (2025)
- [Wormhole Dynamics in Deep Neural Networks](https://mlsanity.com/paper/2508.15086/) (2025)
- [Transferability of Winning Lottery Tickets in Neural Network Differential Equation Solvers](https://mlsanity.com/paper/2306.09863/) (2023)
- [Y-Drop: A Conductance based Dropout for fully connected layers](https://mlsanity.com/paper/2409.09088/) (2024)

## Abstract (arXiv)

This article exposes the failure of some big neural networks to leverage added capacity to reduce underfitting. Past research suggest diminishing returns when increasing the size of neural networks. Our experiments on ImageNet LSVRC-2010 show that this may be due to the fact there are highly diminishing returns for capacity in terms of training error, leading to underfitting. This suggests that the optimization method - first order gradient descent - fails at this regime. Directly attacking this problem, either through the optimization method or the choices of parametrization, may allow to improve the generalization error on large datasets, for which a large capacity is required.