# P-KDGAN: Progressive Knowledge Distillation with GANs for One-class Novelty Detection

arXiv: https://arxiv.org/abs/2007.06963 · HTML: https://mlsanity.com/paper/2007.06963/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 11 of 11)

- [Self-Supervised Masking for Unsupervised Anomaly Detection and Localization](https://mlsanity.com/paper/2205.06568/) (2022)
- [One-Class Classification: A Survey](https://mlsanity.com/paper/2101.03064/) (2021)
- [Adversarially Robust One-class Novelty Detection](https://mlsanity.com/paper/2108.11168/) (2021)
- [Relation Matters: Foreground-aware Graph-based Relational Reasoning for Domain Adaptive Object Detection](https://mlsanity.com/paper/2206.02355/) (2022)
- [Projection Regret: Reducing Background Bias for Novelty Detection via Diffusion Models](https://mlsanity.com/paper/2312.02615/) (2023)
- [BP-Triplet Net for Unsupervised Domain Adaptation: A Bayesian Perspective](https://mlsanity.com/paper/2202.09541/) (2022)
- [Few-shot Fine-tuning is All You Need for Source-free Domain Adaptation](https://mlsanity.com/paper/2304.00792/) (2023)
- [HAD-Net: A Hierarchical Adversarial Knowledge Distillation Network for Improved Enhanced Tumour Segmentation Without Post-Contrast Images](https://mlsanity.com/paper/2103.16617/) (2021)
- [One-Step Forward and Backtrack: Overcoming Zig-Zagging in Loss-Aware Quantization Training](https://mlsanity.com/paper/2401.16760/) (2024)
- [A Survey on GAN Acceleration Using Memory Compression Technique](https://mlsanity.com/paper/2108.06626/) (2021)
- [Cross-Domain Label Propagation for Domain Adaptation with Discriminative Graph Self-Learning](https://mlsanity.com/paper/2302.08710/) (2023)

## Abstract (arXiv)

One-class novelty detection is to identify anomalous instances that do not conform to the expected normal instances. In this paper, the Generative Adversarial Networks (GANs) based on encoder-decoder-encoder pipeline are used for detection and achieve state-of-the-art performance. However, deep neural networks are too over-parameterized to deploy on resource-limited devices. Therefore, Progressive Knowledge Distillation with GANs (PKDGAN) is proposed to learn compact and fast novelty detection networks. The P-KDGAN is a novel attempt to connect two standard GANs by the designed distillation loss for transferring knowledge from the teacher to the student. The progressive learning of knowledge distillation is a two-step approach that continuously improves the performance of the student GAN and achieves better performance than single step methods. In the first step, the student GAN learns the basic knowledge totally from the teacher via guiding of the pretrained teacher GAN with fixed weights. In the second step, joint fine-training is adopted for the knowledgeable teacher and student GANs to further improve the performance and stability. The experimental results on CIFAR-10, MNIST, and FMNIST show that our method improves the performance of the student GAN by 2.44%, 1.77%, and 1.73% when compressing the computation at ratios of 24.45:1, 311.11:1, and 700:1, respectively.