# Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression

arXiv: https://arxiv.org/abs/2012.03096 · HTML: https://mlsanity.com/paper/2012.03096/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2020.

## Built on by (most cited first, 10 of 10)

- [TFormer: A Transmission-Friendly ViT Model for IoT Devices](https://mlsanity.com/paper/2302.07734/) (2023)
- [Pipe-BD: Pipelined Parallel Blockwise Distillation](https://mlsanity.com/paper/2301.12443/) (2023)
- [FedD2S: Personalized Data-Free Federated Knowledge Distillation](https://mlsanity.com/paper/2402.10846/) (2024)
- [Knowledge Distillation: Enhancing Neural Network Compression with Integrated Gradients](https://mlsanity.com/paper/2503.13008/) (2025)
- [Computer Vision Model Compression Techniques for Embedded Systems: A Survey](https://mlsanity.com/paper/2408.08250/) (2024)
- [Onboard Optimization and Learning: A Survey](https://mlsanity.com/paper/2505.08793/) (2025)
- [DONNAv2 -- Lightweight Neural Architecture Search for Vision tasks](https://mlsanity.com/paper/2309.14670/) (2023)
- [Joint Learning for Scattered Point Cloud Understanding with Hierarchical Self-Distillation](https://mlsanity.com/paper/2312.16902/) (2023)
- [Model compression using knowledge distillation with integrated gradients](https://mlsanity.com/paper/2506.14440/) (2025)
- [Layer-wise Curriculum Learning for Efficient LLM Compression](https://mlsanity.com/paper/2609.19213/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) have been extremely successful in solving many challenging AI tasks in natural language processing, speech recognition, and computer vision nowadays. However, DNNs are typically computation intensive, memory demanding, and power hungry, which significantly limits their usage on platforms with constrained resources. Therefore, a variety of compression techniques (e.g. quantization, pruning, and knowledge distillation) have been proposed to reduce the size and power consumption of DNNs. Blockwise knowledge distillation is one of the compression techniques that can effectively reduce the size of a highly complex DNN. However, it is not widely adopted due to its long training time. In this paper, we propose a novel parallel blockwise distillation algorithm to accelerate the distillation process of sophisticated DNNs. Our algorithm leverages local information to conduct independent blockwise distillation, utilizes depthwise separable layers as the efficient replacement block architecture, and properly addresses limiting factors (e.g. dependency, synchronization, and load balancing) that affect parallelism. The experimental results running on an AMD server with four Geforce RTX 2080Ti GPUs show that our algorithm can achieve 3x speedup plus 19% energy savings on VGG distillation, and 3.5x speedup plus 29% energy savings on ResNet distillation, both with negligible accuracy loss. The speedup of ResNet distillation can be further improved to 3.87 when using four RTX6000 GPUs in a distributed cluster.