# Impact of L1 Batch Normalization on Analog Noise Resistant Property of Deep Learning Models

arXiv: https://arxiv.org/abs/2205.04886 · HTML: https://mlsanity.com/paper/2205.04886/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 7 May 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Understanding Batch Normalization](https://mlsanity.com/paper/1806.02375/) (2018)
- [Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models](https://mlsanity.com/paper/1702.03275/) (2017)
- [Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation](https://mlsanity.com/paper/2001.04974/) (2020)
- [L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks](https://mlsanity.com/paper/1802.09769/) (2018)
- [Deep neural networks are robust to weight binarization and other non-linear distortions](https://mlsanity.com/paper/1606.01981/) (2016)
- [Improving Noise Tolerance of Mixed-Signal Neural Networks](https://mlsanity.com/paper/1904.01705/) (2019)
- [Functional Error Correction for Robust Neural Networks](https://mlsanity.com/paper/2001.03814/) (2020)
- [Benchmarking Inference Performance of Deep Learning Models on Analog Devices](https://mlsanity.com/paper/2011.11840/) (2020)
- [A Joint Energy and Latency Framework for Transfer Learning over 5G Industrial Edge Networks](https://mlsanity.com/paper/2104.09382/) (2021)
- [AnalogNets: ML-HW Co-Design of Noise-robust TinyML Models and Always-On Analog Compute-in-Memory Accelerator](https://mlsanity.com/paper/2111.06503/) (2021)
- [Underwater Acoustic Communication Channel Modeling using Deep Learning](https://mlsanity.com/paper/2201.10056/) (2022)
- [Compiling Neural Networks for a Computational Memory Accelerator](https://mlsanity.com/paper/2003.04293/) (2020)

## Abstract (arXiv)

Analog hardware has become a popular choice for machine learning on resource-constrained devices recently due to its fast execution and energy efficiency. However, the inherent presence of noise in analog hardware and the negative impact of the noise on deployed deep neural network (DNN) models limit their usage. The degradation in performance due to the noise calls for the novel design of DNN models that have excellent noiseresistant property, leveraging the properties of the fundamental building block of DNN models. In this work, the use of L1 or TopK BatchNorm type, a fundamental DNN model building block, in designing DNN models with excellent noise-resistant property is proposed. Specifically, a systematic study has been carried out by training DNN models with L1/TopK BatchNorm type, and the performance is compared with DNN models with L2 BatchNorm types. The resulting model noise-resistant property is tested by injecting additive noise to the model weights and evaluating the new model inference accuracy due to the noise. The results show that L1 and TopK BatchNorm type has excellent noise-resistant property, and there is no sacrifice in performance due to the change in the BatchNorm type from L2 to L1/TopK BatchNorm type.