# Model Compression Using Optimal Transport

arXiv: https://arxiv.org/abs/2012.03907 · HTML: https://mlsanity.com/paper/2012.03907/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2020.

## Built on by (most cited first, 8 of 8)

- [Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation](https://mlsanity.com/paper/2412.08139/) (2024)
- [Distilling Privileged Multimodal Information for Expression Recognition using Optimal Transport](https://mlsanity.com/paper/2401.15489/) (2024)
- [Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition](https://mlsanity.com/paper/2408.09035/) (2024)
- [Selective Cross-Task Distillation](https://mlsanity.com/paper/2204.11526/) (2022)
- [LaCoOT: Layer Collapse through Optimal Transport](https://mlsanity.com/paper/2406.08933/) (2024)
- [Identifying Sub-networks in Neural Networks via Functionally Similar Representations](https://mlsanity.com/paper/2410.16484/) (2024)
- [KD2M: A unifying framework for feature knowledge distillation](https://mlsanity.com/paper/2504.01757/) (2025)
- [Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks](https://mlsanity.com/paper/2511.04494/) (2025)

## Abstract (arXiv)

Model compression methods are important to allow for easier deployment of deep learning models in compute, memory and energy-constrained environments such as mobile phones. Knowledge distillation is a class of model compression algorithm where knowledge from a large teacher network is transferred to a smaller student network thereby improving the student's performance. In this paper, we show how optimal transport-based loss functions can be used for training a student network which encourages learning student network parameters that help bring the distribution of student features closer to that of the teacher features. We present image classification results on CIFAR-100, SVHN and ImageNet and show that the proposed optimal transport loss functions perform comparably to or better than other loss functions.