# Noise Optimization for Artificial Neural Networks

arXiv: https://arxiv.org/abs/2102.04450 · HTML: https://mlsanity.com/paper/2102.04450/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 54 references that are in the map.
- First posted on arXiv 6 Feb 2021.

## Builds on (most cited first, 30 of 35)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [Defense against Adversarial Attacks Using High-Level Representation Guided Denoiser](https://mlsanity.com/paper/1712.02976/) (2017)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [Simple Black-box Adversarial Attacks](https://mlsanity.com/paper/1905.07121/) (2019)
- [Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients](https://mlsanity.com/paper/1711.09404/) (2017)
- [Why do deep convolutional networks generalize so poorly to small image transformations?](https://mlsanity.com/paper/1805.12177/) (2018)
- [A study of the effect of JPG compression on adversarial images](https://mlsanity.com/paper/1608.00853/) (2016)
- [Stochastic Activation Pruning for Robust Adversarial Defense](https://mlsanity.com/paper/1803.01442/) (2018)
- [Towards Robust Neural Networks via Random Self-ensemble](https://mlsanity.com/paper/1712.00673/) (2017)
- [The Space of Transferable Adversarial Examples](https://mlsanity.com/paper/1704.03453/) (2017)
- [Improving the Robustness of Deep Neural Networks via Stability Training](https://mlsanity.com/paper/1604.04326/) (2016)
- [Adding Gradient Noise Improves Learning for Very Deep Networks](https://mlsanity.com/paper/1511.06807/) (2015)
- [Examining the Impact of Blur on Recognition by Convolutional Networks](https://mlsanity.com/paper/1611.05760/) (2016)
- [Noisy Activation Functions](https://mlsanity.com/paper/1603.00391/) (2016)
- [What it Thinks is Important is Important: Robustness Transfers through Input Gradients](https://mlsanity.com/paper/1912.05699/) (2019)
- [Cubic LSTMs for Video Prediction](https://mlsanity.com/paper/1904.09412/) (2019)
- [Adversarial Noise Layer: Regularize Neural Network By Adding Noise](https://mlsanity.com/paper/1805.08000/) (2018)
- [Estimate and Replace: A Novel Approach to Integrating Deep Neural Networks with Existing Applications](https://mlsanity.com/paper/1804.09028/) (2018)

## Built on by (most cited first, 7 of 7)

- [One Forward is Enough for Neural Network Training via Likelihood Ratio Method](https://mlsanity.com/paper/2305.08960/) (2023)
- [A Novel Noise Injection-based Training Scheme for Better Model Robustness](https://mlsanity.com/paper/2302.10802/) (2023)
- [Noise Injection Node Regularization for Robust Learning](https://mlsanity.com/paper/2210.15764/) (2022)
- [The Paradox of Noise: An Empirical Study of Noise-Infusion Mechanisms to Improve Generalization, Stability, and Privacy in Federated Learning](https://mlsanity.com/paper/2311.05790/) (2023)
- [Statistical Quality and Reproducibility of Pseudorandom Number Generators in Machine Learning technologies](https://mlsanity.com/paper/2507.03007/) (2025)
- [Random Aggregate Beamforming for Over-the-Air Federated Learning in Large-Scale Networks](https://mlsanity.com/paper/2403.18946/) (2024)
- [Interleaved Noise Injection Improves Clean, Corrupted, and OOD Performance](https://mlsanity.com/paper/2607.14466/) (2026)

## Abstract (arXiv)

Adding noises to artificial neural network(ANN) has been shown to be able to improve robustness in previous work. In this work, we propose a new technique to compute the pathwise stochastic gradient estimate with respect to the standard deviation of the Gaussian noise added to each neuron of the ANN. By our proposed technique, the gradient estimate with respect to noise levels is a byproduct of the backpropagation algorithm for estimating gradient with respect to synaptic weights in ANN. Thus, the noise level for each neuron can be optimized simultaneously in the processing of training the synaptic weights at nearly no extra computational cost. In numerical experiments, our proposed method can achieve significant performance improvement on robustness of several popular ANN structures under both black box and white box attacks tested in various computer vision datasets.