# Effect of Batch Normalization on Noise Resistant Property of Deep Learning Models

arXiv: https://arxiv.org/abs/2205.07372 · HTML: https://mlsanity.com/paper/2205.07372/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 62 references that are in the map.
- First posted on arXiv 15 May 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [Understanding Batch Normalization](https://mlsanity.com/paper/1806.02375/) (2018)
- [Improving the Robustness of Deep Neural Networks via Stability Training](https://mlsanity.com/paper/1604.04326/) (2016)
- [Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models](https://mlsanity.com/paper/1702.03275/) (2017)
- [Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization](https://mlsanity.com/paper/1710.05179/) (2017)
- [SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning](https://mlsanity.com/paper/1805.07898/) (2018)
- [Neuromorphic Hardware In The Loop: Training a Deep Spiking Network on the BrainScaleS Wafer-Scale System](https://mlsanity.com/paper/1703.01909/) (2017)
- [Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation](https://mlsanity.com/paper/2001.04974/) (2020)
- [Multiscale Co-Design Analysis of Energy, Latency, Area, and Accuracy of a ReRAM Analog Neural Training Accelerator](https://mlsanity.com/paper/1707.09952/) (2017)
- [L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks](https://mlsanity.com/paper/1802.09769/) (2018)
- [Deep neural networks are robust to weight binarization and other non-linear distortions](https://mlsanity.com/paper/1606.01981/) (2016)
- [Dynamic Precision Analog Computing for Neural Networks](https://mlsanity.com/paper/2102.06365/) (2021)
- [Improving Noise Tolerance of Mixed-Signal Neural Networks](https://mlsanity.com/paper/1904.01705/) (2019)
- [Functional Error Correction for Robust Neural Networks](https://mlsanity.com/paper/2001.03814/) (2020)
- [Benchmarking Inference Performance of Deep Learning Models on Analog Devices](https://mlsanity.com/paper/2011.11840/) (2020)
- [Device-aware inference operations in SONOS nonvolatile memory arrays](https://mlsanity.com/paper/2004.00802/) (2020)
- [Training Recurrent Neural Networks against Noisy Computations during Inference](https://mlsanity.com/paper/1807.06555/) (2018)
- [Noisy Computations during Inference: Harmful or Helpful?](https://mlsanity.com/paper/1811.10649/) (2018)

## Built on by (most cited first, 2 of 2)

- [Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting](https://mlsanity.com/paper/2410.16150/) (2024)
- [Noise robust neural network architecture](https://mlsanity.com/paper/2305.09276/) (2023)

## Abstract (arXiv)

The fast execution speed and energy efficiency of analog hardware has made them a strong contender for deployment of deep learning model at the edge. However, there are concerns about the presence of analog noise which causes changes to the weight of the models, leading to performance degradation of deep learning model, despite their inherent noise resistant characteristics. The effect of the popular batch normalization layer on the noise resistant ability of deep learning model is investigated in this work. This systematic study has been carried out by first training different models with and without batch normalization layer on CIFAR10 and CIFAR100 dataset. The weights of the resulting models are then injected with analog noise and the performance of the models on the test dataset is obtained and compared. The results show that the presence of batch normalization layer negatively impacts noise resistant property of deep learning model and the impact grows with the increase of the number of batch normalization layers.