# Knowledge distillation using unlabeled mismatched images

arXiv: https://arxiv.org/abs/1703.07131 · HTML: https://mlsanity.com/paper/1703.07131/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Mar 2017.

## Built on by (most cited first, 4 of 4)

- [Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data](https://mlsanity.com/paper/2110.15094/) (2021)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [Scalable Collaborative Learning via Representation Sharing](https://mlsanity.com/paper/2211.10943/) (2022)
- [Knowledge Distillation for Anomaly Detection](https://mlsanity.com/paper/2310.06047/) (2023)

## Abstract (arXiv)

Current approaches for Knowledge Distillation (KD) either directly use training data or sample from the training data distribution. In this paper, we demonstrate effectiveness of 'mismatched' unlabeled stimulus to perform KD for image classification networks. For illustration, we consider scenarios where this is a complete absence of training data, or mismatched stimulus has to be used for augmenting a small amount of training data. We demonstrate that stimulus complexity is a key factor for distillation's good performance. Our examples include use of various datasets for stimulating MNIST and CIFAR teachers.