# CrossNorm and SelfNorm for Generalization under Distribution Shifts

arXiv: https://arxiv.org/abs/2102.02811 · HTML: https://mlsanity.com/paper/2102.02811/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 55 references that are in the map.
- First posted on arXiv 4 Feb 2021.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Improving robustness against common corruptions by covariate shift adaptation](https://mlsanity.com/paper/2006.16971/) (2020)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [Universal Style Transfer via Feature Transforms](https://mlsanity.com/paper/1705.08086/) (2017)

## Built on by (most cited first, 30 of 33)

- [Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization](https://mlsanity.com/paper/2203.07740/) (2022)
- [Visual Prompt Tuning for Test-time Domain Adaptation](https://mlsanity.com/paper/2210.04831/) (2022)
- [Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2312.04265/) (2023)
- [Pin the Memory: Learning to Generalize Semantic Segmentation](https://mlsanity.com/paper/2204.03609/) (2022)
- [Neural Mean Discrepancy for Efficient Out-of-Distribution Detection](https://mlsanity.com/paper/2104.11408/) (2021)
- [SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learning](https://mlsanity.com/paper/2412.04077/) (2024)
- [Normalization Enhances Generalization in Visual Reinforcement Learning](https://mlsanity.com/paper/2306.00656/) (2023)
- [Semantic Self-adaptation: Enhancing Generalization with a Single Sample](https://mlsanity.com/paper/2208.05788/) (2022)
- [Normalization Perturbation: A Simple Domain Generalization Method for Real-World Domain Shifts](https://mlsanity.com/paper/2211.04393/) (2022)
- [Moderately Distributional Exploration for Domain Generalization](https://mlsanity.com/paper/2304.13976/) (2023)
- [LFME: A Simple Framework for Learning from Multiple Experts in Domain Generalization](https://mlsanity.com/paper/2410.17020/) (2024)
- [Stronger, Steadier & Superior: Geometric Consistency in Depth VFM Forges Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2504.12753/) (2025)
- [Generalized Face Anti-spoofing via Finer Domain Partition and Disentangling Liveness-irrelevant Factors](https://mlsanity.com/paper/2407.08243/) (2024)
- [Cross-Domain Few-Shot Classification via Inter-Source Stylization](https://mlsanity.com/paper/2208.08015/) (2022)
- [INSURE: An Information Theory Inspired Disentanglement and Purification Model for Domain Generalization](https://mlsanity.com/paper/2309.04063/) (2023)
- [Cooperative Self-Training for Multi-Target Adaptive Semantic Segmentation](https://mlsanity.com/paper/2210.01578/) (2022)
- [A Novel Cross-Perturbation for Single Domain Generalization](https://mlsanity.com/paper/2308.00918/) (2023)
- [Bridging the Source-to-target Gap for Cross-domain Person Re-Identification with Intermediate Domains](https://mlsanity.com/paper/2203.01682/) (2022)
- [Style Variable and Irrelevant Learning for Generalizable Person Re-identification](https://mlsanity.com/paper/2209.05235/) (2022)
- [Learning Gradient-based Mixup with Extrapolation toward Flatter Minima for Domain Generalization](https://mlsanity.com/paper/2209.14742/) (2022)
- [Learning to Generalize towards Unseen Domains via a Content-Aware Style Invariant Model for Disease Detection from Chest X-rays](https://mlsanity.com/paper/2302.13991/) (2023)
- [Domain Expansion and Boundary Growth for Open-Set Single-Source Domain Generalization](https://mlsanity.com/paper/2411.02920/) (2024)
- [Out-of-Distribution Generalization on Graphs via Progressive Inference](https://mlsanity.com/paper/2503.02988/) (2025)
- [Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models](https://mlsanity.com/paper/2508.01667/) (2025)
- [Mode-Dependent Rectification for Stable PPO Training](https://mlsanity.com/paper/2602.05619/) (2026)
- [A simple normalization technique using window statistics to improve the out-of-distribution generalization on medical images](https://mlsanity.com/paper/2207.03366/) (2022)
- [A Short Survey of Systematic Generalization](https://mlsanity.com/paper/2211.11956/) (2022)
- [Patch-aware Batch Normalization for Improving Cross-domain Robustness](https://mlsanity.com/paper/2304.02848/) (2023)
- [Shape Guided Gradient Voting for Domain Generalization](https://mlsanity.com/paper/2306.10809/) (2023)
- [Boosting Generalization with Adaptive Style Techniques for Fingerprint Liveness Detection](https://mlsanity.com/paper/2310.13573/) (2023)

## Abstract (arXiv)

Traditional normalization techniques (e.g., Batch Normalization and Instance Normalization) generally and simplistically assume that training and test data follow the same distribution. As distribution shifts are inevitable in real-world applications, well-trained models with previous normalization methods can perform badly in new environments. Can we develop new normalization methods to improve generalization robustness under distribution shifts? In this paper, we answer the question by proposing CrossNorm and SelfNorm. CrossNorm exchanges channel-wise mean and variance between feature maps to enlarge training distribution, while SelfNorm uses attention to recalibrate the statistics to bridge gaps between training and test distributions. CrossNorm and SelfNorm can complement each other, though exploring different directions in statistics usage. Extensive experiments on different fields (vision and language), tasks (classification and segmentation), settings (supervised and semi-supervised), and distribution shift types (synthetic and natural) show the effectiveness. Code is available at https://github.com/amazon-research/crossnorm-selfnorm