# KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and Quantization

arXiv: https://arxiv.org/abs/2011.14691 · HTML: https://mlsanity.com/paper/2011.14691/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2020.

## Built on by (most cited first, 3 of 3)

- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Toward Student-Oriented Teacher Network Training For Knowledge Distillation](https://mlsanity.com/paper/2206.06661/) (2022)
- [Real-time Face Mask Detection in Video Data](https://mlsanity.com/paper/2105.01816/) (2021)

## Abstract (arXiv)

In recent years, the growing size of neural networks has led to a vast amount of research concerning compression techniques to mitigate the drawbacks of such large sizes. Most of these research works can be categorized into three broad families : Knowledge Distillation, Pruning, and Quantization. While there has been steady research in this domain, adoption and commercial usage of the proposed techniques has not quite progressed at the rate. We present KD-Lib, an open-source PyTorch based library, which contains state-of-the-art modular implementations of algorithms from the three families on top of multiple abstraction layers. KD-Lib is model and algorithm-agnostic, with extended support for hyperparameter tuning using Optuna and Tensorboard for logging and monitoring. The library can be found at - https://github.com/SforAiDl/KD_Lib.