# SHADE: Information Based Regularization for Deep Learning

arXiv: https://arxiv.org/abs/1804.10988 · HTML: https://mlsanity.com/paper/1804.10988/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2018.

## Built on by (most cited first, 3 of 3)

- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularization](https://mlsanity.com/paper/2208.01134/) (2022)
- [Fidelity Estimation Improves Noisy-Image Classification With Pretrained Networks](https://mlsanity.com/paper/2106.00673/) (2021)

## Abstract (arXiv)

Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define a prior based on conditional entropy, which explicitly decouples the learning of invariant representations in the regularizer and the learning of correlations between inputs and labels in the data fitting term. Our second contribution is to derive a stochastic version of the regularizer compatible with deep learning, resulting in a tractable training scheme. We empirically validate the efficiency of our approach to improve classification performances compared to common regularization schemes on several standard architectures.