# SSN: Learning Sparse Switchable Normalization via SparsestMax

arXiv: https://arxiv.org/abs/1903.03793 · HTML: https://mlsanity.com/paper/1903.03793/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2019.

## Built on by (most cited first, 14 of 14)

- [Adversarially Adaptive Normalization for Single Domain Generalization](https://mlsanity.com/paper/2106.01899/) (2021)
- [Model Agnostic Sample Reweighting for Out-of-Distribution Learning](https://mlsanity.com/paper/2301.09819/) (2023)
- [Probabilistic Bilevel Coreset Selection](https://mlsanity.com/paper/2301.09880/) (2023)
- [Foundation Model is Efficient Multimodal Multitask Model Selector](https://mlsanity.com/paper/2308.06262/) (2023)
- [Trilevel Neural Architecture Search for Efficient Single Image Super-Resolution](https://mlsanity.com/paper/2101.06658/) (2021)
- [OIMNet++: Prototypical Normalization and Localization-aware Learning for Person Search](https://mlsanity.com/paper/2207.10320/) (2022)
- [Dynamic Token Normalization Improves Vision Transformers](https://mlsanity.com/paper/2112.02624/) (2021)
- [Logit Normalization for Long-tail Object Detection](https://mlsanity.com/paper/2203.17020/) (2022)
- [Delving into the Estimation Shift of Batch Normalization in a Network](https://mlsanity.com/paper/2203.10778/) (2022)
- [Unified Normalization for Accelerating and Stabilizing Transformers](https://mlsanity.com/paper/2208.01313/) (2022)
- [Separable Batch Normalization for Robust Facial Landmark Localization with Cross-protocol Network Training](https://mlsanity.com/paper/2101.06663/) (2021)
- [L2BN: Enhancing Batch Normalization by Equalizing the L2 Norms of Features](https://mlsanity.com/paper/2207.02625/) (2022)
- [A simple normalization technique using window statistics to improve the out-of-distribution generalization on medical images](https://mlsanity.com/paper/2207.03366/) (2022)
- [Unified Batch Normalization: Identifying and Alleviating the Feature Condensation in Batch Normalization and a Unified Framework](https://mlsanity.com/paper/2311.15993/) (2023)

## Abstract (arXiv)

Normalization methods improve both optimization and generalization of ConvNets. To further boost performance, the recently-proposed switchable normalization (SN) provides a new perspective for deep learning: it learns to select different normalizers for different convolution layers of a ConvNet. However, SN uses softmax function to learn importance ratios to combine normalizers, leading to redundant computations compared to a single normalizer. This work addresses this issue by presenting Sparse Switchable Normalization (SSN) where the importance ratios are constrained to be sparse. Unlike ℓ1 and ℓ0 constraints that impose difficulties in optimization, we turn this constrained optimization problem into feed-forward computation by proposing SparsestMax, which is a sparse version of softmax. SSN has several appealing properties. (1) It inherits all benefits from SN such as applicability in various tasks and robustness to a wide range of batch sizes. (2) It is guaranteed to select only one normalizer for each normalization layer, avoiding redundant computations. (3) SSN can be transferred to various tasks in an end-to-end manner. Extensive experiments show that SSN outperforms its counterparts on various challenging benchmarks such as ImageNet, Cityscapes, ADE20K, and Kinetics.