# Adversarial Network Compression

arXiv: https://arxiv.org/abs/1803.10750 · HTML: https://mlsanity.com/paper/1803.10750/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2018.

## Built on by (most cited first, 9 of 9)

- [Complementary Relation Contrastive Distillation](https://mlsanity.com/paper/2103.16367/) (2021)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [Lightweight Monocular Depth Estimation through Guided Decoding](https://mlsanity.com/paper/2203.04206/) (2022)
- [LANA: Latency Aware Network Acceleration](https://mlsanity.com/paper/2107.10624/) (2021)
- [HEMP: High-order Entropy Minimization for neural network comPression](https://mlsanity.com/paper/2107.05298/) (2021)
- [Visual Domain Adaptation for Monocular Depth Estimation on Resource-Constrained Hardware](https://mlsanity.com/paper/2108.02671/) (2021)
- [Aligning Logits Generatively for Principled Black-Box Knowledge Distillation](https://mlsanity.com/paper/2205.10490/) (2022)
- [Compact CNN Structure Learning by Knowledge Distillation](https://mlsanity.com/paper/2104.09191/) (2021)
- [Feature Representation Transferring to Lightweight Models via Perception Coherence](https://mlsanity.com/paper/2505.06595/) (2025)

## Abstract (arXiv)

Neural network compression has recently received much attention due to the computational requirements of modern deep models. In this work, our objective is to transfer knowledge from a deep and accurate model to a smaller one. Our contributions are threefold: (i) we propose an adversarial network compression approach to train the small student network to mimic the large teacher, without the need for labels during training; (ii) we introduce a regularization scheme to prevent a trivially-strong discriminator without reducing the network capacity and (iii) our approach generalizes on different teacher-student models. In an extensive evaluation on five standard datasets, we show that our student has small accuracy drop, achieves better performance than other knowledge transfer approaches and it surpasses the performance of the same network trained with labels. In addition, we demonstrate state-of-the-art results compared to other compression strategies.