# Regularizing by the Variance of the Activations' Sample-Variances

arXiv: https://arxiv.org/abs/1811.08764 · HTML: https://mlsanity.com/paper/1811.08764/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2018.

## Built on by (most cited first, 5 of 5)

- [ACAT: Adversarial Counterfactual Attention for Classification and Detection in Medical Imaging](https://mlsanity.com/paper/2303.15421/) (2023)
- [Training Deep Neural Networks with Joint Quantization and Pruning of Weights and Activations](https://mlsanity.com/paper/2110.08271/) (2021)
- [Neuron with Steady Response Leads to Better Generalization](https://mlsanity.com/paper/2111.15414/) (2021)
- [EqDeepRx: Learning a Scalable and Interference Mitigating MIMO Receiver](https://mlsanity.com/paper/2602.11834/) (2026)
- [An Adaptive Method Stabilizing Activations for Enhanced Generalization](https://mlsanity.com/paper/2506.08353/) (2025)

## Abstract (arXiv)

Normalization techniques play an important role in supporting efficient and often more effective training of deep neural networks. While conventional methods explicitly normalize the activations, we suggest to add a loss term instead. This new loss term encourages the variance of the activations to be stable and not vary from one random mini-batch to the next. As we prove, this encourages the activations to be distributed around a few distinct modes. We also show that if the inputs are from a mixture of two Gaussians, the new loss would either join the two together, or separate between them optimally in the LDA sense, depending on the prior probabilities. Finally, we are able to link the new regularization term to the batchnorm method, which provides it with a regularization perspective. Our experiments demonstrate an improvement in accuracy over the batchnorm technique for both CNNs and fully connected networks.