# Resiliency of Deep Neural Networks under Quantization

arXiv: https://arxiv.org/abs/1511.06488 · HTML: https://mlsanity.com/paper/1511.06488/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2015.

## Built on by (most cited first, 30 of 35)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [PFGM++: Unlocking the Potential of Physics-Inspired Generative Models](https://mlsanity.com/paper/2302.04265/) (2023)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Efficient Post-training Quantization with FP8 Formats](https://mlsanity.com/paper/2309.14592/) (2023)
- [Lifting the Curse of Capacity Gap in Distilling Language Models](https://mlsanity.com/paper/2305.12129/) (2023)
- [Quantifying the Capabilities of LLMs across Scale and Precision](https://mlsanity.com/paper/2405.03146/) (2024)
- [Generative Zero-shot Network Quantization](https://mlsanity.com/paper/2101.08430/) (2021)
- [SPIQ: Data-Free Per-Channel Static Input Quantization](https://mlsanity.com/paper/2203.14642/) (2022)
- [Diverse Sample Generation: Pushing the Limit of Generative Data-free Quantization](https://mlsanity.com/paper/2109.00212/) (2021)
- [NeuralDEM -- Real-time Simulation of Industrial Particulate Flows](https://mlsanity.com/paper/2411.09678/) (2024)
- [MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity](https://mlsanity.com/paper/2407.20021/) (2024)
- [Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications](https://mlsanity.com/paper/2508.15008/) (2025)
- [Random and Adversarial Bit Error Robustness: Energy-Efficient and Secure DNN Accelerators](https://mlsanity.com/paper/2104.08323/) (2021)
- [A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statistics](https://mlsanity.com/paper/2112.02834/) (2021)
- [Neural Networks with Quantization Constraints](https://mlsanity.com/paper/2210.15623/) (2022)
- [Hyperspherical Quantization: Toward Smaller and More Accurate Models](https://mlsanity.com/paper/2212.12653/) (2022)
- [Low-latency machine learning FPGA accelerator for multi-qubit-state discrimination](https://mlsanity.com/paper/2407.03852/) (2024)
- [Reduced-Order Neural Network Synthesis with Robustness Guarantees](https://mlsanity.com/paper/2102.09284/) (2021)
- [Elastic-Link for Binarized Neural Network](https://mlsanity.com/paper/2112.10149/) (2021)
- [FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Training](https://mlsanity.com/paper/2211.10948/) (2022)
- [FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation](https://mlsanity.com/paper/2203.12091/) (2022)
- [Ternary Quantization: A Survey](https://mlsanity.com/paper/2303.01505/) (2023)
- [Variance-Aware Noisy Training: Hardening DNNs against Unstable Analog Computations](https://mlsanity.com/paper/2503.16183/) (2025)
- [Adaptive Mesh-Quantization for Neural PDE Solvers](https://mlsanity.com/paper/2511.18474/) (2025)
- [Latency-Constrained DNN Architecture Learning for Edge Systems using Zerorized Batch Normalization](https://mlsanity.com/paper/2607.06922/) (2026)
- [MobileNMT: Enabling Translation in 15MB and 30ms](https://mlsanity.com/paper/2306.04235/) (2023)
- [Regularization-based Framework for Quantization-, Fault- and Variability-Aware Training](https://mlsanity.com/paper/2503.01297/) (2025)
- [Q-Rater: Non-Convex Optimization for Post-Training Uniform Quantization](https://mlsanity.com/paper/2105.01868/) (2021)
- [A Construction Kit for Efficient Low Power Neural Network Accelerator Designs](https://mlsanity.com/paper/2106.12810/) (2021)

## Abstract (arXiv)

The complexity of deep neural network algorithms for hardware implementation can be much lowered by optimizing the word-length of weights and signals. Direct quantization of floating-point weights, however, does not show good performance when the number of bits assigned is small. Retraining of quantized networks has been developed to relieve this problem. In this work, the effects of retraining are analyzed for a feedforward deep neural network (FFDNN) and a convolutional neural network (CNN). The network complexity is controlled to know their effects on the resiliency of quantized networks by retraining. The complexity of the FFDNN is controlled by varying the unit size in each hidden layer and the number of layers, while that of the CNN is done by modifying the feature map configuration. We find that the performance gap between the floating-point and the retrain-based ternary (+1, 0, -1) weight neural networks exists with a fair amount in 'complexity limited' networks, but the discrepancy almost vanishes in fully complex networks whose capability is limited by the training data, rather than by the number of connections. This research shows that highly complex DNNs have the capability of absorbing the effects of severe weight quantization through retraining, but connection limited networks are less resilient. This paper also presents the effective compression ratio to guide the trade-off between the network size and the precision when the hardware resource is limited.