# Gradient ℓ1 Regularization for Quantization Robustness

arXiv: https://arxiv.org/abs/2002.07520 · HTML: https://mlsanity.com/paper/2002.07520/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2020.

## Built on by (most cited first, 4 of 4)

- [GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization](https://mlsanity.com/paper/2210.06895/) (2022)
- [Update Compression for Deep Neural Networks on the Edge](https://mlsanity.com/paper/2203.04516/) (2022)
- [One Model for All Quantization: A Quantized Network Supporting Hot-Swap Bit-Width Adjustment](https://mlsanity.com/paper/2105.01353/) (2021)
- [Adversarial Parameter Defense by Multi-Step Risk Minimization](https://mlsanity.com/paper/2109.02889/) (2021)

## Abstract (arXiv)

We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-demand to different bit-widths as energy and memory requirements of the application change. Unlike quantization-aware training using the straight-through estimator that only targets a specific bit-width and requires access to training data and pipeline, our regularization-based method paves the way for "on the fly'' post-training quantization to various bit-widths. We show that by modeling quantization as a ℓ_∞-bounded perturbation, the first-order term in the loss expansion can be regularized using the ℓ1-norm of gradients. We experimentally validate the effectiveness of our regularization scheme on different architectures on CIFAR-10 and ImageNet datasets.