# Attentive Normalization

arXiv: https://arxiv.org/abs/1908.01259 · HTML: https://mlsanity.com/paper/1908.01259/

## Key facts

- Cited by 13 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 4 Aug 2019.

## Built on by (most cited first, 13 of 13)

- [Learning Auxiliary Monocular Contexts Helps Monocular 3D Object Detection](https://mlsanity.com/paper/2112.04628/) (2021)
- [CrossNorm and SelfNorm for Generalization under Distribution Shifts](https://mlsanity.com/paper/2102.02811/) (2021)
- [Learning Local-Global Contextual Adaptation for Multi-Person Pose Estimation](https://mlsanity.com/paper/2109.03622/) (2021)
- [OIMNet++: Prototypical Normalization and Localization-aware Learning for Person Search](https://mlsanity.com/paper/2207.10320/) (2022)
- [A Survey of Robust 3D Object Detection Methods in Point Clouds](https://mlsanity.com/paper/2204.00106/) (2022)
- [Sandwich Batch Normalization: A Drop-In Replacement for Feature Distribution Heterogeneity](https://mlsanity.com/paper/2102.11382/) (2021)
- [STC speaker recognition systems for the NIST SRE 2021](https://mlsanity.com/paper/2111.02298/) (2021)
- [Decentralised Person Re-Identification with Selective Knowledge Aggregation](https://mlsanity.com/paper/2110.11384/) (2021)
- [Revisiting Batch Norm Initialization](https://mlsanity.com/paper/2110.13989/) (2021)
- [Separable Batch Normalization for Robust Facial Landmark Localization with Cross-protocol Network Training](https://mlsanity.com/paper/2101.06663/) (2021)
- [SO(2) and O(2) Equivariance in Image Recognition with Bessel-Convolutional Neural Networks](https://mlsanity.com/paper/2304.09214/) (2023)
- [A simple normalization technique using window statistics to improve the out-of-distribution generalization on medical images](https://mlsanity.com/paper/2207.03366/) (2022)
- [Efficient Feature Aggregation and Scale-Aware Regression for Monocular 3D Object Detection](https://mlsanity.com/paper/2411.02747/) (2024)

## Abstract (arXiv)

In state-of-the-art deep neural networks, both feature normalization and feature attention have become ubiquitous. % with significant performance improvement shown in a vast amount of tasks. They are usually studied as separate modules, however. In this paper, we propose a light-weight integration between the two schema and present Attentive Normalization (AN). Instead of learning a single affine transformation, AN learns a mixture of affine transformations and utilizes their weighted-sum as the final affine transformation applied to re-calibrate features in an instance-specific way. The weights are learned by leveraging channel-wise feature attention. In experiments, we test the proposed AN using four representative neural architectures in the ImageNet-1000 classification benchmark and the MS-COCO 2017 object detection and instance segmentation benchmark. AN obtains consistent performance improvement for different neural architectures in both benchmarks with absolute increase of top-1 accuracy in ImageNet-1000 between 0.5% and 2.7%, and absolute increase up to 1.8% and 2.2% for bounding box and mask AP in MS-COCO respectively. We observe that the proposed AN provides a strong alternative to the widely used Squeeze-and-Excitation (SE) module. The source codes are publicly available at https://github.com/iVMCL/AOGNet-v2 (the ImageNet Classification Repo) and https://github.com/iVMCL/AttentiveNormDetection (the MS-COCO Detection and Segmentation Repo).