# torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation

arXiv: https://arxiv.org/abs/2011.12913 · HTML: https://mlsanity.com/paper/2011.12913/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2020.

## Built on by (most cited first, 19 of 19)

- [Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery](https://mlsanity.com/paper/2206.10540/) (2022)
- [Knowledge Distillation Based on Transformed Teacher Matching](https://mlsanity.com/paper/2402.11148/) (2024)
- [Supervised Compression for Resource-Constrained Edge Computing Systems](https://mlsanity.com/paper/2108.11898/) (2021)
- [Understanding the Role of the Projector in Knowledge Distillation](https://mlsanity.com/paper/2303.11098/) (2023)
- [Information Theoretic Representation Distillation](https://mlsanity.com/paper/2112.00459/) (2021)
- [SC2 Benchmark: Supervised Compression for Split Computing](https://mlsanity.com/paper/2203.08875/) (2022)
- [Good Teachers Explain: Explanation-Enhanced Knowledge Distillation](https://mlsanity.com/paper/2402.03119/) (2024)
- [FrankenSplit: Efficient Neural Feature Compression with Shallow Variational Bottleneck Injection for Mobile Edge Computing](https://mlsanity.com/paper/2302.10681/) (2023)
- [On the Impact of Knowledge Distillation for Model Interpretability](https://mlsanity.com/paper/2305.15734/) (2023)
- [Applications of Knowledge Distillation in Remote Sensing: A Survey](https://mlsanity.com/paper/2409.12111/) (2024)
- [Toward Student-Oriented Teacher Network Training For Knowledge Distillation](https://mlsanity.com/paper/2206.06661/) (2022)
- [FOOL: Addressing the Downlink Bottleneck in Satellite Computing with Neural Feature Compression](https://mlsanity.com/paper/2403.16677/) (2024)
- [Improving the Reproducibility of Deep Learning Software: An Initial Investigation through a Case Study Analysis](https://mlsanity.com/paper/2505.03165/) (2025)
- [Knowledge Distillation from Single to Multi Labels: an Empirical Study](https://mlsanity.com/paper/2303.08360/) (2023)
- [Architectural Vision for Quantum Computing in the Edge-Cloud Continuum](https://mlsanity.com/paper/2305.05238/) (2023)
- [torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP](https://mlsanity.com/paper/2310.17644/) (2023)
- [Smaller3d: Smaller Models for 3D Semantic Segmentation Using Minkowski Engine and Knowledge Distillation Methods](https://mlsanity.com/paper/2305.03188/) (2023)
- [Structural and Statistical Audio Texture Knowledge Distillation for Acoustic Classification](https://mlsanity.com/paper/2501.01921/) (2025)
- [Optimizing Teacher-Student Partitioning for Scalable Knowledge Distillation on HPC Systems](https://mlsanity.com/paper/2606.27797/) (2026)

## Abstract (arXiv)

While knowledge distillation (transfer) has been attracting attentions from the research community, the recent development in the fields has heightened the need for reproducible studies and highly generalized frameworks to lower barriers to such high-quality, reproducible deep learning research. Several researchers voluntarily published frameworks used in their knowledge distillation studies to help other interested researchers reproduce their original work. Such frameworks, however, are usually neither well generalized nor maintained, thus researchers are still required to write a lot of code to refactor/build on the frameworks for introducing new methods, models, datasets and designing experiments. In this paper, we present our developed open-source framework built on PyTorch and dedicated for knowledge distillation studies. The framework is designed to enable users to design experiments by declarative PyYAML configuration files, and helps researchers complete the recently proposed ML Code Completeness Checklist. Using the developed framework, we demonstrate its various efficient training strategies, and implement a variety of knowledge distillation methods. We also reproduce some of their original experimental results on the ImageNet and COCO datasets presented at major machine learning conferences such as ICLR, NeurIPS, CVPR and ECCV, including recent state-of-the-art methods. All the source code, configurations, log files and trained model weights are publicly available at https://github.com/yoshitomo-matsubara/torchdistill .