# Deep neural networks are robust to weight binarization and other non-linear distortions

arXiv: https://arxiv.org/abs/1606.01981 · HTML: https://mlsanity.com/paper/1606.01981/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2016.

## Built on by (most cited first, 13 of 13)

- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices](https://mlsanity.com/paper/2103.05579/) (2021)
- [Ps and Qs: Quantization-aware pruning for efficient low latency neural network inference](https://mlsanity.com/paper/2102.11289/) (2021)
- [Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis](https://mlsanity.com/paper/2104.09396/) (2021)
- [Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment](https://mlsanity.com/paper/2111.06862/) (2021)
- [QReg: On Regularization Effects of Quantization](https://mlsanity.com/paper/2206.12372/) (2022)
- [Random and Adversarial Bit Error Robustness: Energy-Efficient and Secure DNN Accelerators](https://mlsanity.com/paper/2104.08323/) (2021)
- [R2F: A Remote Retraining Framework for AIoT Processors with Computing Errors](https://mlsanity.com/paper/2107.03096/) (2021)
- [T3DNet: Compressing Point Cloud Models for Lightweight 3D Recognition](https://mlsanity.com/paper/2402.19264/) (2024)
- [Effect of Batch Normalization on Noise Resistant Property of Deep Learning Models](https://mlsanity.com/paper/2205.07372/) (2022)
- [Clipped DeepControl: deep neural network two-dimensional pulse design with an amplitude constraint layer](https://mlsanity.com/paper/2201.08668/) (2022)
- [Impact of L1 Batch Normalization on Analog Noise Resistant Property of Deep Learning Models](https://mlsanity.com/paper/2205.04886/) (2022)
- [Impact of Learning Rate on Noise Resistant Property of Deep Learning Models](https://mlsanity.com/paper/2205.07856/) (2022)

## Abstract (arXiv)

Recent results show that deep neural networks achieve excellent performance even when, during training, weights are quantized and projected to a binary representation. Here, we show that this is just the tip of the iceberg: these same networks, during testing, also exhibit a remarkable robustness to distortions beyond quantization, including additive and multiplicative noise, and a class of non-linear projections where binarization is just a special case. To quantify this robustness, we show that one such network achieves 11% test error on CIFAR-10 even with 0.68 effective bits per weight. Furthermore, we find that a common training heuristic--namely, projecting quantized weights during backpropagation--can be altered (or even removed) and networks still achieve a base level of robustness during testing. Specifically, training with weight projections other than quantization also works, as does simply clipping the weights, both of which have never been reported before. We confirm our results for CIFAR-10 and ImageNet datasets. Finally, drawing from these ideas, we propose a stochastic projection rule that leads to a new state of the art network with 7.64% test error on CIFAR-10 using no data augmentation.