# Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation

arXiv: https://arxiv.org/abs/2001.04974 · HTML: https://mlsanity.com/paper/2001.04974/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2020.

## Built on by (most cited first, 17 of 17)

- [Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators](https://mlsanity.com/paper/2302.08469/) (2023)
- [Neural Network Compression for Noisy Storage Devices](https://mlsanity.com/paper/2102.07725/) (2021)
- [Quantum-limited stochastic optical neural networks operating at a few quanta per activation](https://mlsanity.com/paper/2307.15712/) (2023)
- [Denoising Noisy Neural Networks: A Bayesian Approach with Compensation](https://mlsanity.com/paper/2105.10699/) (2021)
- [Analog Foundation Models](https://mlsanity.com/paper/2505.09663/) (2025)
- [Effect of Batch Normalization on Noise Resistant Property of Deep Learning Models](https://mlsanity.com/paper/2205.07372/) (2022)
- [Walking Noise: On Layer-Specific Robustness of Neural Architectures against Noisy Computations and Associated Characteristic Learning Dynamics](https://mlsanity.com/paper/2212.10430/) (2022)
- [One-Shot Online Testing of Deep Neural Networks Based on Distribution Shift Detection](https://mlsanity.com/paper/2305.09348/) (2023)
- [Variance-Aware Noisy Training: Hardening DNNs against Unstable Analog Computations](https://mlsanity.com/paper/2503.16183/) (2025)
- [Sensitivity-Aware Finetuning for Accuracy Recovery on Deep Learning Hardware](https://mlsanity.com/paper/2306.03076/) (2023)
- [Variability-Aware Training and Self-Tuning of Highly Quantized DNNs for Analog PIM](https://mlsanity.com/paper/2111.06457/) (2021)
- [Impact of L1 Batch Normalization on Analog Noise Resistant Property of Deep Learning Models](https://mlsanity.com/paper/2205.04886/) (2022)
- [Impact of Learning Rate on Noise Resistant Property of Deep Learning Models](https://mlsanity.com/paper/2205.07856/) (2022)
- [Implications of Noise in Resistive Memory on Deep Neural Networks for Image Classification](https://mlsanity.com/paper/2401.05820/) (2024)
- [Few-Shot Testing: Estimating Uncertainty of Memristive Deep Neural Networks Using One Bayesian Test Vector](https://mlsanity.com/paper/2405.18894/) (2024)
- [Beyond Noise: Understanding and Overcoming Temperature Effects in Analog DNN Inference](https://mlsanity.com/paper/2609.15527/) (2026)
- [Depth Laws for the Precision Floor of Trained Neural Networks: Amplification, Residual Scaling, and a Quantization-Aware Training Paradox](https://mlsanity.com/paper/2609.32060/) (2026)

## Abstract (arXiv)

The success of deep learning has brought forth a wave of interest in computer hardware design to better meet the high demands of neural network inference. In particular, analog computing hardware has been heavily motivated specifically for accelerating neural networks, based on either electronic, optical or photonic devices, which may well achieve lower power consumption than conventional digital electronics. However, these proposed analog accelerators suffer from the intrinsic noise generated by their physical components, which makes it challenging to achieve high accuracy on deep neural networks. Hence, for successful deployment on analog accelerators, it is essential to be able to train deep neural networks to be robust to random continuous noise in the network weights, which is a somewhat new challenge in machine learning. In this paper, we advance the understanding of noisy neural networks. We outline how a noisy neural network has reduced learning capacity as a result of loss of mutual information between its input and output. To combat this, we propose using knowledge distillation combined with noise injection during training to achieve more noise robust networks, which is demonstrated experimentally across different networks and datasets, including ImageNet. Our method achieves models with as much as two times greater noise tolerance compared with the previous best attempts, which is a significant step towards making analog hardware practical for deep learning.