# Exemplar Normalization for Learning Deep Representation

arXiv: https://arxiv.org/abs/2003.08761 · HTML: https://mlsanity.com/paper/2003.08761/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2020.

## Built on by (most cited first, 6 of 6)

- [CrossNorm and SelfNorm for Generalization under Distribution Shifts](https://mlsanity.com/paper/2102.02811/) (2021)
- [Weight Conditioning for Smooth Optimization of Neural Networks](https://mlsanity.com/paper/2409.03424/) (2024)
- [Logit Normalization for Long-tail Object Detection](https://mlsanity.com/paper/2203.17020/) (2022)
- [Delving into the Estimation Shift of Batch Normalization in a Network](https://mlsanity.com/paper/2203.10778/) (2022)
- [Imagined Autocurricula](https://mlsanity.com/paper/2509.13341/) (2025)
- [SIRe-Networks: Convolutional Neural Networks Architectural Extension for Information Preservation via Skip/Residual Connections and Interlaced Auto-Encoders](https://mlsanity.com/paper/2110.02776/) (2021)

## Abstract (arXiv)

Normalization techniques are important in different advanced neural networks and different tasks. This work investigates a novel dynamic learning-to-normalize (L2N) problem by proposing Exemplar Normalization (EN), which is able to learn different normalization methods for different convolutional layers and image samples of a deep network. EN significantly improves flexibility of the recently proposed switchable normalization (SN), which solves a static L2N problem by linearly combining several normalizers in each normalization layer (the combination is the same for all samples). Instead of directly employing a multi-layer perceptron (MLP) to learn data-dependent parameters as conditional batch normalization (cBN) did, the internal architecture of EN is carefully designed to stabilize its optimization, leading to many appealing benefits. (1) EN enables different convolutional layers, image samples, categories, benchmarks, and tasks to use different normalization methods, shedding light on analyzing them in a holistic view. (2) EN is effective for various network architectures and tasks. (3) It could replace any normalization layers in a deep network and still produce stable model training. Extensive experiments demonstrate the effectiveness of EN in a wide spectrum of tasks including image recognition, noisy label learning, and semantic segmentation. For example, by replacing BN in the ordinary ResNet50, improvement produced by EN is 300% more than that of SN on both ImageNet and the noisy WebVision dataset.