# Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise

arXiv: https://arxiv.org/abs/1908.04008 · HTML: https://mlsanity.com/paper/1908.04008/

## Key facts

- Cited by 28 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 12 Aug 2019.

## Built on by (most cited first, 28 of 28)

- [RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs](https://mlsanity.com/paper/2503.10657/) (2025)
- [CrossNorm and SelfNorm for Generalization under Distribution Shifts](https://mlsanity.com/paper/2102.02811/) (2021)
- [ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connection](https://mlsanity.com/paper/2310.13545/) (2023)
- [Blending Pruning Criteria for Convolutional Neural Networks](https://mlsanity.com/paper/2107.05033/) (2021)
- [MoExtend: Tuning New Experts for Modality and Task Extension](https://mlsanity.com/paper/2408.03511/) (2024)
- [On Fast Simulation of Dynamical System with Neural Vector Enhanced Numerical Solver](https://mlsanity.com/paper/2208.03680/) (2022)
- [Understanding Self-attention Mechanism via Dynamical System Perspective](https://mlsanity.com/paper/2308.09939/) (2023)
- [The Lottery Ticket Hypothesis for Self-attention in Convolutional Neural Network](https://mlsanity.com/paper/2207.07858/) (2022)
- [MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models](https://mlsanity.com/paper/2505.19959/) (2025)
- [AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training](https://mlsanity.com/paper/2107.05804/) (2021)
- [Mirror Gradient: Towards Robust Multimodal Recommender Systems via Exploring Flat Local Minima](https://mlsanity.com/paper/2402.11262/) (2024)
- [A Generic Shared Attention Mechanism for Various Backbone Neural Networks](https://mlsanity.com/paper/2210.16101/) (2022)
- [Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks](https://mlsanity.com/paper/2305.15912/) (2023)
- [Switchable Self-attention Module](https://mlsanity.com/paper/2209.05680/) (2022)
- [AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarios](https://mlsanity.com/paper/2302.10184/) (2023)
- [LSAS: Lightweight Sub-attention Strategy for Alleviating Attention Bias Problem](https://mlsanity.com/paper/2305.05200/) (2023)
- [Revisiting Batch Norm Initialization](https://mlsanity.com/paper/2110.13989/) (2021)
- [Mix-Pooling Strategy for Attention Mechanism](https://mlsanity.com/paper/2208.10322/) (2022)
- [NoMorelization: Building Normalizer-Free Models from a Sample's Perspective](https://mlsanity.com/paper/2210.06932/) (2022)
- [ASR: Attention-alike Structural Re-parameterization](https://mlsanity.com/paper/2304.06345/) (2023)
- [Separable Batch Normalization for Robust Facial Landmark Localization with Cross-protocol Network Training](https://mlsanity.com/paper/2101.06663/) (2021)
- [Efficient Climate Simulation via Machine Learning Method](https://mlsanity.com/paper/2209.08151/) (2022)
- [Causal Inference for Chatting Handoff](https://mlsanity.com/paper/2210.02862/) (2022)
- [Deepening Neural Networks Implicitly and Locally via Recurrent Attention Strategy](https://mlsanity.com/paper/2210.15676/) (2022)
- [LayerAct: Advanced Activation Mechanism for Robust Inference of CNNs](https://mlsanity.com/paper/2306.04940/) (2023)
- [Unified Batch Normalization: Identifying and Alleviating the Feature Condensation in Batch Normalization and a Unified Framework](https://mlsanity.com/paper/2311.15993/) (2023)
- [Incremental Object Keypoint Learning](https://mlsanity.com/paper/2503.20248/) (2025)
- [Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes](https://mlsanity.com/paper/2511.17399/) (2025)

## Abstract (arXiv)

Batch Normalization (BN)(Ioffe and Szegedy 2015) normalizes the features of an input image via statistics of a batch of images and hence BN will bring the noise to the gradient of the training loss. Previous works indicate that the noise is important for the optimization and generalization of deep neural networks, but too much noise will harm the performance of networks. In our paper, we offer a new point of view that self-attention mechanism can help to regulate the noise by enhancing instance-specific information to obtain a better regularization effect. Therefore, we propose an attention-based BN called Instance Enhancement Batch Normalization (IEBN) that recalibrates the information of each channel by a simple linear transformation. IEBN has a good capacity of regulating noise and stabilizing network training to improve generalization even in the presence of two kinds of noise attacks during training. Finally, IEBN outperforms BN with only a light parameter increment in image classification tasks for different network structures and benchmark datasets.