# Additive Noise Annealing and Approximation Properties of Quantized Neural Networks

arXiv: https://arxiv.org/abs/1905.10452 · HTML: https://mlsanity.com/paper/1905.10452/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 5 of 5)

- [Sound Event Detection with Binary Neural Networks on Tightly Power-Constrained IoT Devices](https://mlsanity.com/paper/2101.04446/) (2021)
- [Sub-mW Keyword Spotting on an MCU: Analog Binary Feature Extraction and Binary Neural Networks](https://mlsanity.com/paper/2201.03386/) (2022)
- [Towards Efficient In-memory Computing Hardware for Quantized Neural Networks: State-of-the-art, Open Challenges and Perspectives](https://mlsanity.com/paper/2307.03936/) (2023)
- [Universal Approximation Theorems of Fully Connected Binarized Neural Networks](https://mlsanity.com/paper/2102.02631/) (2021)
- [Synaptic metaplasticity with multi-level memristive devices](https://mlsanity.com/paper/2306.12142/) (2023)

## Abstract (arXiv)

We present a theoretical and experimental investigation of the quantization problem for artificial neural networks. We provide a mathematical definition of quantized neural networks and analyze their approximation capabilities, showing in particular that any Lipschitz-continuous map defined on a hypercube can be uniformly approximated by a quantized neural network. We then focus on the regularization effect of additive noise on the arguments of multi-step functions inherent to the quantization of continuous variables. In particular, when the expectation operator is applied to a non-differentiable multi-step random function, and if the underlying probability density is differentiable (in either classical or weak sense), then a differentiable function is retrieved, with explicit bounds on its Lipschitz constant. Based on these results, we propose a novel gradient-based training algorithm for quantized neural networks that generalizes the straight-through estimator, acting on noise applied to the network's parameters. We evaluate our algorithm on the CIFAR-10 and ImageNet image classification benchmarks, showing state-of-the-art performance on AlexNet and MobileNetV2 for ternary networks.