# Denoising Noisy Neural Networks: A Bayesian Approach with Compensation

arXiv: https://arxiv.org/abs/2105.10699 · HTML: https://mlsanity.com/paper/2105.10699/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 60 references that are in the map.
- First posted on arXiv 22 May 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [Federated Learning via Over-the-Air Computation](https://mlsanity.com/paper/1812.11750/) (2018)
- [Broadband Analog Aggregation for Low-Latency Federated Edge Learning (Extended Version)](https://mlsanity.com/paper/1812.11494/) (2018)
- [Federated Learning over Wireless Fading Channels](https://mlsanity.com/paper/1907.09769/) (2019)
- [One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysis](https://mlsanity.com/paper/2001.05713/) (2020)
- [Training Convolutional Networks with Noisy Labels](https://mlsanity.com/paper/1406.2080/) (2014)
- [Blind Federated Edge Learning](https://mlsanity.com/paper/2010.10030/) (2020)
- [Deep Learning for mmWave Beam and Blockage Prediction Using Sub-6GHz Channels](https://mlsanity.com/paper/1910.02900/) (2019)
- [Machine Learning in the Air](https://mlsanity.com/paper/1904.12385/) (2019)
- [Communicate to Learn at the Edge](https://mlsanity.com/paper/2009.13269/) (2020)
- [Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks](https://mlsanity.com/paper/2001.05674/) (2020)
- [Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation](https://mlsanity.com/paper/2001.04974/) (2020)
- [Federated mmWave Beam Selection Utilizing LIDAR Data](https://mlsanity.com/paper/2102.02802/) (2021)
- [Neural Network Compression for Noisy Storage Devices](https://mlsanity.com/paper/2102.07725/) (2021)
- [Benchmarking Inference Performance of Deep Learning Models on Analog Devices](https://mlsanity.com/paper/2011.11840/) (2020)
- [Gaussian Approximation of Quantization Error for Estimation from Compressed Data](https://mlsanity.com/paper/2001.03243/) (2020)
- [Information contraction in noisy binary neural networks and its implications](https://mlsanity.com/paper/2101.11750/) (2021)

## Built on by (most cited first, 4 of 4)

- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [AirNet: Neural Network Transmission over the Air](https://mlsanity.com/paper/2105.11166/) (2021)
- [Purification Of Contaminated Convolutional Neural Networks Via Robust Recovery: An Approach with Theoretical Guarantee in One-Hidden-Layer Case](https://mlsanity.com/paper/2407.11031/) (2024)
- [DP-TTA: Test-time Adaptation for Transient Electromagnetic Signal Denoising via Dictionary-driven Prior Regularization](https://mlsanity.com/paper/2510.13160/) (2025)

## Abstract (arXiv)

Deep neural networks (DNNs) with noisy weights, which we refer to as noisy neural networks (NoisyNNs), arise from the training and inference of DNNs in the presence of noise. NoisyNNs emerge in many new applications, including the wireless transmission of DNNs, the efficient deployment or storage of DNNs in analog devices, and the truncation or quantization of DNN weights. This paper studies a fundamental problem of NoisyNNs: how to reconstruct the DNN weights from their noisy manifestations. While all prior works relied on the maximum likelihood (ML) estimation, this paper puts forth a denoising approach to reconstruct DNNs with the aim of maximizing the inference accuracy of the reconstructed models. The superiority of our denoiser is rigorously proven in two small-scale problems, wherein we consider a quadratic neural network function and a shallow feedforward neural network, respectively. When applied to advanced learning tasks with modern DNN architectures, our denoiser exhibits significantly better performance than the ML estimator. Consider the average test accuracy of the denoised DNN model versus the weight variance to noise power ratio (WNR) performance. When denoising a noisy ResNet34 model arising from noisy inference, our denoiser outperforms ML estimation by up to 4.1 dB to achieve a test accuracy of 60%.When denoising a noisy ResNet18 model arising from noisy training, our denoiser outperforms ML estimation by 13.4 dB and 8.3 dB to achieve test accuracies of 60% and 80%, respectively.