# Understanding and mitigating noise in trained deep neural networks

arXiv: https://arxiv.org/abs/2103.07413 · HTML: https://mlsanity.com/paper/2103.07413/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2021.

## Built on by (most cited first, 9 of 9)

- [An optical neural network using less than 1 photon per multiplication](https://mlsanity.com/paper/2104.13467/) (2021)
- [Quantum-limited stochastic optical neural networks operating at a few quanta per activation](https://mlsanity.com/paper/2307.15712/) (2023)
- [Noise mitigation strategies in physical feedforward neural networks](https://mlsanity.com/paper/2204.09461/) (2022)
- [Hybrid Quantum-inspired Resnet and Densenet for Pattern Recognition](https://mlsanity.com/paper/2403.05754/) (2024)
- [On Learnable Parameters of Optimal and Suboptimal Deep Learning Models](https://mlsanity.com/paper/2408.11720/) (2024)
- [Impact of white noise in artificial neural networks trained for classification: performance and noise mitigation strategies](https://mlsanity.com/paper/2411.04354/) (2024)
- [An Empirical Study on Fault Detection and Root Cause Analysis of Indium Tin Oxide Electrodes by Processing S-parameter Patterns](https://mlsanity.com/paper/2308.11639/) (2023)
- [Learning Galaxy Intrinsic Alignment Correlations](https://mlsanity.com/paper/2404.13702/) (2024)
- [Impact of white Gaussian internal noise on analog echo-state neural networks](https://mlsanity.com/paper/2405.07670/) (2024)

## Abstract (arXiv)

Deep neural networks unlocked a vast range of new applications by solving tasks of which many were previously deemed as reserved to higher human intelligence. One of the developments enabling this success was a boost in computing power provided by special purpose hardware, such as graphic or tensor processing units. However, these do not leverage fundamental features of neural networks like parallelism and analog state variables. Instead, they emulate neural networks relying on binary computing, which results in unsustainable energy consumption and comparatively low speed. Fully parallel and analogue hardware promises to overcome these challenges, yet the impact of analogue neuron noise and its propagation, i.e. accumulation, threatens rendering such approaches inept. Here, we determine for the first time the propagation of noise in deep neural networks comprising noisy nonlinear neurons in trained fully connected layers. We study additive and multiplicative as well as correlated and uncorrelated noise, and develop analytical methods that predict the noise level in any layer of symmetric deep neural networks or deep neural networks trained with back propagation. We find that noise accumulation is generally bound, and adding additional network layers does not worsen the signal to noise ratio beyond a limit. Most importantly, noise accumulation can be suppressed entirely when neuron activation functions have a slope smaller than unity. We therefore developed the framework for noise in fully connected deep neural networks implemented in analog systems, and identify criteria allowing engineers to design noise-resilient novel neural network hardware.