# Feature-map-level Online Adversarial Knowledge Distillation

arXiv: https://arxiv.org/abs/2002.01775 · HTML: https://mlsanity.com/paper/2002.01775/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2020.

## Built on by (most cited first, 30 of 62)

- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [A Fast Knowledge Distillation Framework for Visual Recognition](https://mlsanity.com/paper/2112.01528/) (2021)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation](https://mlsanity.com/paper/2103.08273/) (2021)
- [NORM: Knowledge Distillation via N-to-One Representation Matching](https://mlsanity.com/paper/2305.13803/) (2023)
- [Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image Classification](https://mlsanity.com/paper/2210.03664/) (2022)
- [Multi-domain Learning for Updating Face Anti-spoofing Models](https://mlsanity.com/paper/2208.11148/) (2022)
- [A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2310.20494/) (2023)
- [Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition](https://mlsanity.com/paper/2207.11518/) (2022)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Student Network Learning via Evolutionary Knowledge Distillation](https://mlsanity.com/paper/2103.13811/) (2021)
- [Switchable Online Knowledge Distillation](https://mlsanity.com/paper/2209.04996/) (2022)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [Distilling a Powerful Student Model via Online Knowledge Distillation](https://mlsanity.com/paper/2103.14473/) (2021)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [Coaching a Teachable Student](https://mlsanity.com/paper/2306.10014/) (2023)
- [Compressing Deep Graph Neural Networks via Adversarial Knowledge Distillation](https://mlsanity.com/paper/2205.11678/) (2022)
- [SelfReg-UNet: Self-Regularized UNet for Medical Image Segmentation](https://mlsanity.com/paper/2406.14896/) (2024)
- [Self-Distilled Self-Supervised Representation Learning](https://mlsanity.com/paper/2111.12958/) (2021)
- [Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distribution](https://mlsanity.com/paper/2109.03075/) (2021)
- [Tiny Object Tracking: A Large-scale Dataset and A Baseline](https://mlsanity.com/paper/2202.05659/) (2022)
- [Weakly Supervised Monocular 3D Detection with a Single-View Image](https://mlsanity.com/paper/2402.19144/) (2024)
- [A Comprehensive Review of Knowledge Distillation in Computer Vision](https://mlsanity.com/paper/2404.00936/) (2024)
- [Dual Discriminator Adversarial Distillation for Data-free Model Compression](https://mlsanity.com/paper/2104.05382/) (2021)
- [Not All Knowledge Is Created Equal: Mutual Distillation of Confident Knowledge](https://mlsanity.com/paper/2106.01489/) (2021)
- [Performance-aware Approximation of Global Channel Pruning for Multitask CNNs](https://mlsanity.com/paper/2303.11923/) (2023)
- [BD-KD: Balancing the Divergences for Online Knowledge Distillation](https://mlsanity.com/paper/2212.12965/) (2022)
- [Illumination Distillation Framework for Nighttime Person Re-Identification and A New Benchmark](https://mlsanity.com/paper/2308.16486/) (2023)
- [CNN-Transformer Rectified Collaborative Learning for Medical Image Segmentation](https://mlsanity.com/paper/2408.13698/) (2024)

## Abstract (arXiv)

Feature maps contain rich information about image intensity and spatial correlation. However, previous online knowledge distillation methods only utilize the class probabilities. Thus in this paper, we propose an online knowledge distillation method that transfers not only the knowledge of the class probabilities but also that of the feature map using the adversarial training framework. We train multiple networks simultaneously by employing discriminators to distinguish the feature map distributions of different networks. Each network has its corresponding discriminator which discriminates the feature map from its own as fake while classifying that of the other network as real. By training a network to fool the corresponding discriminator, it can learn the other network's feature map distribution. We show that our method performs better than the conventional direct alignment method such as L1 and is more suitable for online distillation. Also, we propose a novel cyclic learning scheme for training more than two networks together. We have applied our method to various network architectures on the classification task and discovered a significant improvement of performance especially in the case of training a pair of a small network and a large one.