mlsanityData to 14 Oct 2026

PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation

arXiv 2106.14681 ↗First posted 25 Jun 2021cs.LGcited by 21

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

As edge devices become prevalent, deploying Deep Neural Networks (DNN) on edge devices has become a critical issue. However, DNN requires a high computational resource which is rarely available for edge devices. To handle this, we propose a novel model compression method for the devices with limited computational resources, called PQK consisting of pruning, quantization, and knowledge distillation (KD) processes. Unlike traditional pruning and KD, PQK makes use of unimportant weights pruned in the pruning process to make a teacher network for training a better student network without pre-training the teacher model. PQK has two phases. Phase 1 exploits iterative pruning and quantization-aware training to make a lightweight and power-efficient model. In phase 2, we make a teacher network by adding unimportant weights unused in phase 1 to a pruned network. By using this teacher network, we train the pruned network as a student network. In doing so, we do not need a pre-trained teacher network for the KD framework because the teacher and the student networks coexist within the same network. We apply our method to the recognition model and verify the effectiveness of PQK on keyword spotting (KWS) and image recognition.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 17 of its 25 references that are in the map. First posted on arXiv 25 Jun 2021.

Builds on

17 of 25 references.

References · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  6. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 367
  7. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  8. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  9. 2020Dynamic Model Pruning with Feedbackcited by 125
  10. 2018NVIDIA Tensor Core Programmability, Performance & Precisioncited by 79
  11. 2017Deep Residual Learning for Small-Footprint Keyword Spottingcited by 68
  12. 2019QKD: Quantization-aware Knowledge Distillationcited by 44
5 more
  1. 2019Feature Fusion for Online Mutual Knowledge Distillationcited by 40
  2. 2020Accurate Detection of Wake Word Start and End Using a CNNcited by 24
  3. 2020Listen Attentively, and Spell Once: Whole Sentence Generation via a Non-Autoregressive Architecture for Low-Latency Speech Recognitioncited by 22
  4. 2020Hybrid Transformer/CTC Networks for Hardware Efficient Voice Triggeringcited by 8
  5. 2021Prototype-based Personalized Pruningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 4; 2024: 5; 2025: 5; 2026: 4.

21 papers.

Sort:

By area:

  1. 2023Model-as-a-Service (MaaS): A Surveycited by 25
  2. 2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
  3. 2023A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Servicescited by 14
  4. 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognitioncited by 11
  5. 2023Hardware-Aware DNN Compression via Diverse Pruning and Mixed-Precision Quantizationcited by 10
  6. 2022Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognitioncited by 5
  7. 2024Outlier Reduction with Gated Attention for Improved Post-training Quantization in Large Sequence-to-sequence Speech Foundation Modelscited by 3
  8. 2026Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compressioncited by 3
  9. 2022Feature-domain Adaptive Contrastive Distillation for Efficient Single Image Super-Resolutioncited by 1
  10. 2024Infrared Domain Adaptation with Zero-Shot Quantizationcited by 1
  11. 2024On the Impact of White-box Deployment Strategies for Edge AI on Latency and Model Performancecited by 1
  12. 2025Dynamic Acoustic Model Architecture Optimization in Training for ASRcited by 1
9 more
  1. 2026Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compressioncited by 1
  2. 2021Performance optimizations on deep noise suppression modelscited by 0
  3. 2024Lightweight Geometric Deep Learning for Molecular Modelling in Catalyst Discoverycited by 0
  4. 2025GeLaCo: An Evolutionary Approach to Layer Compressioncited by 0
  5. 2025Beyond Imaging: Vision Transformer Digital Twin Surrogates for 3D+T Biological Tissue Dynamicscited by 0
  6. 2025Integrating Pruning with Quantization for Efficient Deep Neural Networks Compressioncited by 0
  7. 2025TinyEcoWeedNet: Edge Efficient Real-Time Aerial Agricultural Weed Detectioncited by 0
  8. 2026CLIP-RD: Relational Distillation for Efficient CLIP Knowledge Distillationcited by 0
  9. 2026Lightweight Vision Transformer Compression for On-Device Plant Disease Detection in Resource-Constrained Agricultural Field Conditionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022OPQ: Compressing Deep Neural Networks with One-shot Pruning-Quantization
  2. 2026Hybrid Compression: Integrating Pruning and Quantization for Optimized Neural Networks
  3. 2024EPSD: Early Pruning with Self-Distillation for Efficient Model Compression
  4. 2021Deep Neural Compression Via Concurrent Pruning and Self-Distillation
  5. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Survey
  6. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
  7. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
  8. 2024Edge AI: Evaluation of Model Compression Techniques for Convolutional Neural Networks