# Differentiable Learning-to-Normalize via Switchable Normalization

arXiv: https://arxiv.org/abs/1806.10779 · HTML: https://mlsanity.com/paper/1806.10779/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2018.

## Built on by (most cited first, 30 of 33)

- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [Adversarially Adaptive Normalization for Single Domain Generalization](https://mlsanity.com/paper/2106.01899/) (2021)
- [Continual Normalization: Rethinking Batch Normalization for Online Continual Learning](https://mlsanity.com/paper/2203.16102/) (2022)
- [Efficient Personalized Federated Learning via Sparse Model-Adaptation](https://mlsanity.com/paper/2305.02776/) (2023)
- [Membrane Potential Batch Normalization for Spiking Neural Networks](https://mlsanity.com/paper/2308.08359/) (2023)
- [Unsupervised Multi-Source Domain Adaptation for Person Re-Identification](https://mlsanity.com/paper/2104.12961/) (2021)
- [pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning](https://mlsanity.com/paper/2402.01350/) (2024)
- [New Insights for the Stability-Plasticity Dilemma in Online Continual Learning](https://mlsanity.com/paper/2302.08741/) (2023)
- [RDP-Net: Region Detail Preserving Network for Change Detection](https://mlsanity.com/paper/2202.09745/) (2022)
- [Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence](https://mlsanity.com/paper/2106.03743/) (2021)
- [Understanding the Failure of Batch Normalization for Transformers in NLP](https://mlsanity.com/paper/2210.05153/) (2022)
- [Generalized Domain Adaptation](https://mlsanity.com/paper/2106.01656/) (2021)
- [Efficiently Assemble Normalization Layers and Regularization for Federated Domain Generalization](https://mlsanity.com/paper/2403.15605/) (2024)
- [Highlighting the Importance of Reducing Research Bias and Carbon Emissions in CNNs](https://mlsanity.com/paper/2106.03242/) (2021)
- [Domain Generalization Guided by Gradient Signal to Noise Ratio of Parameters](https://mlsanity.com/paper/2310.07361/) (2023)
- [Semantic Self-adaptation: Enhancing Generalization with a Single Sample](https://mlsanity.com/paper/2208.05788/) (2022)
- [Delving into the Estimation Shift of Batch Normalization in a Network](https://mlsanity.com/paper/2203.10778/) (2022)
- [Unified Normalization for Accelerating and Stabilizing Transformers](https://mlsanity.com/paper/2208.01313/) (2022)
- [Reducing the feature divergence of RGB and near-infrared images using Switchable Normalization](https://mlsanity.com/paper/2106.03088/) (2021)
- [Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks](https://mlsanity.com/paper/2305.15912/) (2023)
- [Deep Learning for Spatio-Temporal Fusion in Land Surface Temperature Estimation: A Comprehensive Survey, Experimental Analysis, and Future Trends](https://mlsanity.com/paper/2412.16631/) (2024)
- [Reciprocal Normalization for Domain Adaptation](https://mlsanity.com/paper/2112.10474/) (2021)
- [Multi-task Geometric Estimation of Depth and Surface Normal from Monocular 360deg Images](https://mlsanity.com/paper/2411.01749/) (2024)
- [AFN: Adaptive Fusion Normalization via an Encoder-Decoder Framework](https://mlsanity.com/paper/2308.03321/) (2023)
- [CCSI: Continual Class-Specific Impression for Data-free Class Incremental Learning](https://mlsanity.com/paper/2406.05631/) (2024)
- [Delving into Variance Transmission and Normalization: Shift of Average Gradient Makes the Network Collapse](https://mlsanity.com/paper/2103.11590/) (2021)
- [Cross-modal Local Shortest Path and Global Enhancement for Visible-Thermal Person Re-Identification](https://mlsanity.com/paper/2206.04401/) (2022)
- [L2BN: Enhancing Batch Normalization by Equalizing the L2 Norms of Features](https://mlsanity.com/paper/2207.02625/) (2022)
- [Theoretical Insight into Batch Normalization: Data Dependant Auto-Tuning of Regularization Rate](https://mlsanity.com/paper/2209.07587/) (2022)
- [An Empirical Analysis of the Shift and Scale Parameters in BatchNorm](https://mlsanity.com/paper/2303.12818/) (2023)

## Abstract (arXiv)

We address a learning-to-normalize problem by proposing Switchable Normalization (SN), which learns to select different normalizers for different normalization layers of a deep neural network. SN employs three distinct scopes to compute statistics (means and variances) including a channel, a layer, and a minibatch. SN switches between them by learning their importance weights in an end-to-end manner. It has several good properties. First, it adapts to various network architectures and tasks (see Fig.1). Second, it is robust to a wide range of batch sizes, maintaining high performance even when small minibatch is presented (e.g. 2 images/GPU). Third, SN does not have sensitive hyper-parameter, unlike group normalization that searches the number of groups as a hyper-parameter. Without bells and whistles, SN outperforms its counterparts on various challenging benchmarks, such as ImageNet, COCO, CityScapes, ADE20K, and Kinetics. Analyses of SN are also presented. We hope SN will help ease the usage and understand the normalization techniques in deep learning. The code of SN has been made available in https://github.com/switchablenorms/.