# KTAN: Knowledge Transfer Adversarial Network

arXiv: https://arxiv.org/abs/1810.08126 · HTML: https://mlsanity.com/paper/1810.08126/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Oct 2018.

## Built on by (most cited first, 4 of 4)

- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [Joint Answering and Explanation for Visual Commonsense Reasoning](https://mlsanity.com/paper/2202.12626/) (2022)
- [Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression](https://mlsanity.com/paper/2104.03164/) (2021)
- [New Perspective on Progressive GANs Distillation for One-class Novelty Detection](https://mlsanity.com/paper/2109.07295/) (2021)

## Abstract (arXiv)

To reduce the large computation and storage cost of a deep convolutional neural network, the knowledge distillation based methods have pioneered to transfer the generalization ability of a large (teacher) deep network to a light-weight (student) network. However, these methods mostly focus on transferring the probability distribution of the softmax layer in a teacher network and thus neglect the intermediate representations. In this paper, we propose a knowledge transfer adversarial network to better train a student network. Our technique holistically considers both intermediate representations and probability distributions of a teacher network. To transfer the knowledge of intermediate representations, we set high-level teacher feature maps as a target, toward which the student feature maps are trained. Specifically, we arrange a Teacher-to-Student layer for enabling our framework suitable for various student structures. The intermediate representation helps the student network better understand the transferred generalization as compared to the probability distribution only. Furthermore, we infuse an adversarial learning process by employing a discriminator network, which can fully exploit the spatial correlation of feature maps in training a student network. The experimental results demonstrate that the proposed method can significantly improve the performance of a student network on both image classification and object detection tasks.