# Data-Free Network Quantization With Adversarial Knowledge Distillation

arXiv: https://arxiv.org/abs/2005.04136 · HTML: https://mlsanity.com/paper/2005.04136/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2020.

## Built on by (most cited first, 30 of 80)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [Diversifying Sample Generation for Accurate Data-Free Quantization](https://mlsanity.com/paper/2103.01049/) (2021)
- [SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation](https://mlsanity.com/paper/2202.07471/) (2022)
- [IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantization](https://mlsanity.com/paper/2111.09136/) (2021)
- [Few-shot Backdoor Defense Using Shapley Estimation](https://mlsanity.com/paper/2112.14889/) (2021)
- [Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples](https://mlsanity.com/paper/2111.02625/) (2021)
- [Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data](https://mlsanity.com/paper/2108.05698/) (2021)
- [Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data](https://mlsanity.com/paper/2110.15094/) (2021)
- [Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay](https://mlsanity.com/paper/2106.09835/) (2021)
- [FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models](https://mlsanity.com/paper/2109.03775/) (2021)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation](https://mlsanity.com/paper/2209.10359/) (2022)
- [Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation](https://mlsanity.com/paper/2302.14290/) (2023)
- [Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training](https://mlsanity.com/paper/2206.06501/) (2022)
- [NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation](https://mlsanity.com/paper/2310.00258/) (2023)
- [DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture](https://mlsanity.com/paper/2409.03550/) (2024)
- [Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?](https://mlsanity.com/paper/2305.12954/) (2023)
- [It's All In the Teacher: Zero-Shot Quantization Brought Closer to the Teacher](https://mlsanity.com/paper/2203.17008/) (2022)
- [Small Scale Data-Free Knowledge Distillation](https://mlsanity.com/paper/2406.07876/) (2024)
- [Integer-only Zero-shot Quantization for Efficient Speech Recognition](https://mlsanity.com/paper/2103.16827/) (2021)
- [Few-Shot Unlearning by Model Inversion](https://mlsanity.com/paper/2205.15567/) (2022)
- [QuantFace: Towards Lightweight Face Recognition by Synthetic Data Low-bit Quantization](https://mlsanity.com/paper/2206.10526/) (2022)
- [Jumping through Local Minima: Quantization in the Loss Landscape of Vision Transformers](https://mlsanity.com/paper/2308.10814/) (2023)
- [AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compression](https://mlsanity.com/paper/2105.12151/) (2021)
- [Generative Zero-shot Network Quantization](https://mlsanity.com/paper/2101.08430/) (2021)
- [CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharing](https://mlsanity.com/paper/2205.11845/) (2022)
- [A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services](https://mlsanity.com/paper/2304.14271/) (2023)
- [Diverse Sample Generation: Pushing the Limit of Generative Data-free Quantization](https://mlsanity.com/paper/2109.00212/) (2021)
- [Towards Data-Free Domain Generalization](https://mlsanity.com/paper/2110.04545/) (2021)

## Abstract (arXiv)

Network quantization is an essential procedure in deep learning for development of efficient fixed-point inference models on mobile or edge platforms. However, as datasets grow larger and privacy regulations become stricter, data sharing for model compression gets more difficult and restricted. In this paper, we consider data-free network quantization with synthetic data. The synthetic data are generated from a generator, while no data are used in training the generator and in quantization. To this end, we propose data-free adversarial knowledge distillation, which minimizes the maximum distance between the outputs of the teacher and the (quantized) student for any adversarial samples from a generator. To generate adversarial samples similar to the original data, we additionally propose matching statistics from the batch normalization layers for generated data and the original data in the teacher. Furthermore, we show the gain of producing diverse adversarial samples by using multiple generators and multiple students. Our experiments show the state-of-the-art data-free model compression and quantization results for (wide) residual networks and MobileNet on SVHN, CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. The accuracy losses compared to using the original datasets are shown to be very minimal.