mlsanityData to 7 Oct 2026

Impact of L1 Batch Normalization on Analog Noise Resistant Property of Deep Learning Models

arXiv 2205.04886 ↗First posted 7 May 2022cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Analog hardware has become a popular choice for machine learning on resource-constrained devices recently due to its fast execution and energy efficiency. However, the inherent presence of noise in analog hardware and the negative impact of the noise on deployed deep neural network (DNN) models limit their usage. The degradation in performance due to the noise calls for the novel design of DNN models that have excellent noiseresistant property, leveraging the properties of the fundamental building block of DNN models. In this work, the use of L1 or TopK BatchNorm type, a fundamental DNN model building block, in designing DNN models with excellent noise-resistant property is proposed. Specifically, a systematic study has been carried out by training DNN models with L1/TopK BatchNorm type, and the performance is compared with DNN models with L2 BatchNorm types. The resulting model noise-resistant property is tested by injecting additive noise to the model weights and evaluating the new model inference accuracy due to the noise. The results show that L1 and TopK BatchNorm type has excellent noise-resistant property, and there is no sacrifice in performance due to the change in the BatchNorm type from L2 to L1/TopK BatchNorm type.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 42 references that are in the map. First posted on arXiv 7 May 2022.

Builds on

23 of 42 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2018Group Normalizationcited by 1,514
  8. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  9. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  10. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  11. 2018How Does Batch Normalization Help Optimization?cited by 355
  12. 2018Understanding Batch Normalizationcited by 157
11 more
  1. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 127
  2. 2020Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillationcited by 17
  3. 2018L1-Norm Batch Normalization for Efficient Training of Deep Neural Networkscited by 15
  4. 2016Deep neural networks are robust to weight binarization and other non-linear distortionscited by 13
  5. 2019Improving Noise Tolerance of Mixed-Signal Neural Networkscited by 8
  6. 2020Benchmarking Inference Performance of Deep Learning Models on Analog Devicescited by 7
  7. 2021A Joint Energy and Latency Framework for Transfer Learning over 5G Industrial Edge Networkscited by 7
  8. 2021AnalogNets: ML-HW Co-Design of Noise-robust TinyML Models and Always-On Analog Compute-in-Memory Acceleratorcited by 7
  9. 2020Functional Error Correction for Robust Neural Networkscited by 6
  10. 2022Underwater Acoustic Communication Channel Modeling using Deep Learningcited by 3
  11. 2020Compiling Neural Networks for a Computational Memory Acceleratorcited by 1

Built on by

Not cited yet.