# Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks

arXiv: https://arxiv.org/abs/2009.14502 · HTML: https://mlsanity.com/paper/2009.14502/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2020.

## Built on by (most cited first, 21 of 21)

- [Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models](https://mlsanity.com/paper/2503.06027/) (2025)
- [Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networks](https://mlsanity.com/paper/2210.16103/) (2022)
- [Quantized Feature Distillation for Network Quantization](https://mlsanity.com/paper/2307.10638/) (2023)
- [FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation](https://mlsanity.com/paper/2407.07093/) (2024)
- [Poster: Self-Supervised Quantization-Aware Knowledge Distillation](https://mlsanity.com/paper/2309.13220/) (2023)
- [Low-bit Model Quantization for Deep Neural Networks: A Survey](https://mlsanity.com/paper/2505.05530/) (2025)
- [Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpart](https://mlsanity.com/paper/2412.15846/) (2024)
- [Enabling All In-Edge Deep Learning: A Literature Review](https://mlsanity.com/paper/2204.03326/) (2022)
- [Vertical Layering of Quantized Neural Networks for Heterogeneous Inference](https://mlsanity.com/paper/2212.05326/) (2022)
- [Differentiable Search for Finding Optimal Quantization Strategy](https://mlsanity.com/paper/2404.08010/) (2024)
- [Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizer](https://mlsanity.com/paper/2509.20854/) (2025)
- [On the Impact of White-box Deployment Strategies for Edge AI on Latency and Model Performance](https://mlsanity.com/paper/2411.00907/) (2024)
- [Binary Neural Networks for Large Language Model: A Survey](https://mlsanity.com/paper/2502.19008/) (2025)
- [Data-Augmented Quantization-Aware Knowledge Distillation](https://mlsanity.com/paper/2509.03850/) (2025)
- [Decoder-Free Distillation for Quantized Image Restoration](https://mlsanity.com/paper/2603.09624/) (2026)
- [SynQ: Accurate Zero-shot Quantization by Synthesis-aware Fine-tuning](https://mlsanity.com/paper/2603.18423/) (2026)
- [Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method](https://mlsanity.com/paper/2306.06625/) (2023)
- [Self-Supervised Quantization-Aware Knowledge Distillation](https://mlsanity.com/paper/2403.11106/) (2024)
- [Conditional Pseudo-Supervised Contrast for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2510.03375/) (2025)
- [Q2: Quantization-Aware Gradient Balancing and Attention Alignment for Low-Bit Quantization](https://mlsanity.com/paper/2511.05898/) (2025)
- [Progressive Multi-Ancestor Bit-Depth Distillation](https://mlsanity.com/paper/2610.04100/) (2026)

## Abstract (arXiv)

The quantization of deep neural networks (QDNNs) has been actively studied for deployment in edge devices. Recent studies employ the knowledge distillation (KD) method to improve the performance of quantized networks. In this study, we propose stochastic precision ensemble training for QDNNs (SPEQ). SPEQ is a knowledge distillation training scheme; however, the teacher is formed by sharing the model parameters of the student network. We obtain the soft labels of the teacher by changing the bit precision of the activation stochastically at each layer of the forward-pass computation. The student model is trained with these soft labels to reduce the activation quantization noise. The cosine similarity loss is employed, instead of the KL-divergence, for KD training. As the teacher model changes continuously by random bit-precision assignment, it exploits the effect of stochastic ensemble KD. SPEQ outperforms the existing quantization training methods in various tasks, such as image classification, question-answering, and transfer learning without the need for cumbersome teacher networks.