# The Effect of Network Width on the Performance of Large-batch Training

arXiv: https://arxiv.org/abs/1806.03791 · HTML: https://mlsanity.com/paper/1806.03791/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2018.

## Built on by (most cited first, 5 of 5)

- [When and Why Momentum Accelerates SGD:An Empirical Study](https://mlsanity.com/paper/2306.09000/) (2023)
- [Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates](https://mlsanity.com/paper/2103.02351/) (2021)
- [Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers](https://mlsanity.com/paper/2411.00999/) (2024)
- [A New Perspective for Understanding Generalization Gap of Deep Neural Networks Trained with Large Batch Sizes](https://mlsanity.com/paper/2210.12184/) (2022)
- [Scaling Laws and Pathologies of Single-Layer PINNs: Network Width and PDE Nonlinearity](https://mlsanity.com/paper/2603.12556/) (2026)

## Abstract (arXiv)

Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however, large batches can affect the convergence properties and generalization performance of SGD. In this work, we take a first step towards analyzing how the structure (width and depth) of a neural network affects the performance of large-batch training. We present new theoretical results which suggest that--for a fixed number of parameters--wider networks are more amenable to fast large-batch training compared to deeper ones. We provide extensive experiments on residual and fully-connected neural networks which suggest that wider networks can be trained using larger batches without incurring a convergence slow-down, unlike their deeper variants.