# MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudes

arXiv: https://arxiv.org/abs/1804.05965 · HTML: https://mlsanity.com/paper/1804.05965/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2018.

## Built on by (most cited first, 3 of 3)

- [Convolutional Dynamic Alignment Networks for Interpretable Classifications](https://mlsanity.com/paper/2104.00032/) (2021)
- [Neuron with Steady Response Leads to Better Generalization](https://mlsanity.com/paper/2111.15414/) (2021)
- [Nonlinearity Enhanced Adaptive Activation Functions](https://mlsanity.com/paper/2403.19896/) (2024)

## Abstract (arXiv)

Effective regularisation of neural networks is essential to combat overfitting due to the large number of parameters involved. We present an empirical analogue to the Lipschitz constant of a feed-forward neural network, which we refer to as the maximum gain. We hypothesise that constraining the gain of a network will have a regularising effect, similar to how constraining the Lipschitz constant of a network has been shown to improve generalisation. A simple algorithm is provided that involves rescaling the weight matrix of each layer after each parameter update. We conduct a series of studies on common benchmark datasets, and also a novel dataset that we introduce to enable easier significance testing for experiments using convolutional networks. Performance on these datasets compares favourably with other common regularisation techniques.