# The Knowledge Within: Methods for Data-Free Model Compression

arXiv: https://arxiv.org/abs/1912.01274 · HTML: https://mlsanity.com/paper/1912.01274/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2019.

## Built on by (most cited first, 30 of 54)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [See through Gradients: Image Batch Recovery via GradInversion](https://mlsanity.com/paper/2104.07586/) (2021)
- [Source-Free Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2103.16372/) (2021)
- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [PD-Quant: Post-Training Quantization based on Prediction Difference Metric](https://mlsanity.com/paper/2212.07048/) (2022)
- [Diversifying Sample Generation for Accurate Data-Free Quantization](https://mlsanity.com/paper/2103.01049/) (2021)
- [IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantization](https://mlsanity.com/paper/2111.09136/) (2021)
- [Few-shot Backdoor Defense Using Shapley Estimation](https://mlsanity.com/paper/2112.14889/) (2021)
- [A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Tasks](https://mlsanity.com/paper/2311.07784/) (2023)
- [Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay](https://mlsanity.com/paper/2201.03019/) (2022)
- [Visualizing Adapted Knowledge in Domain Transfer](https://mlsanity.com/paper/2104.10602/) (2021)
- [Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data](https://mlsanity.com/paper/2108.05698/) (2021)
- [Applications and Techniques for Fast Machine Learning in Science](https://mlsanity.com/paper/2110.13041/) (2021)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?](https://mlsanity.com/paper/2305.12954/) (2023)
- [It's All In the Teacher: Zero-Shot Quantization Brought Closer to the Teacher](https://mlsanity.com/paper/2203.17008/) (2022)
- [Integer-only Zero-shot Quantization for Efficient Speech Recognition](https://mlsanity.com/paper/2103.16827/) (2021)
- [Differentially Private Knowledge Distillation via Synthetic Text Generation](https://mlsanity.com/paper/2403.00932/) (2024)
- [AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compression](https://mlsanity.com/paper/2105.12151/) (2021)
- [Genie: Show Me the Data for Quantization](https://mlsanity.com/paper/2212.04780/) (2022)
- [IDEAL: Query-Efficient Data-Free Learning from Black-box Models](https://mlsanity.com/paper/2205.11158/) (2022)
- [Generative Zero-shot Network Quantization](https://mlsanity.com/paper/2101.08430/) (2021)
- [PrivateSNN: Privacy-Preserving Spiking Neural Networks](https://mlsanity.com/paper/2104.03414/) (2021)
- [CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharing](https://mlsanity.com/paper/2205.11845/) (2022)
- [Hard Sample Matters a Lot in Zero-Shot Quantization](https://mlsanity.com/paper/2303.13826/) (2023)
- [ZipNN: Lossless Compression for AI Models](https://mlsanity.com/paper/2411.05239/) (2024)
- [Self-calibration for Language Model Quantization and Pruning](https://mlsanity.com/paper/2410.17170/) (2024)
- [Diverse Sample Generation: Pushing the Limit of Generative Data-free Quantization](https://mlsanity.com/paper/2109.00212/) (2021)
- [Dual Discriminator Adversarial Distillation for Data-free Model Compression](https://mlsanity.com/paper/2104.05382/) (2021)
- [Fine-grained Data Distribution Alignment for Post-Training Quantization](https://mlsanity.com/paper/2109.04186/) (2021)

## Abstract (arXiv)

Recently, an extensive amount of research has been focused on compressing and accelerating Deep Neural Networks (DNN). So far, high compression rate algorithms require part of the training dataset for a low precision calibration, or a fine-tuning process. However, this requirement is unacceptable when the data is unavailable or contains sensitive information, as in medical and biometric use-cases. We present three methods for generating synthetic samples from trained models. Then, we demonstrate how these samples can be used to calibrate and fine-tune quantized models without using any real data in the process. Our best performing method has a negligible accuracy degradation compared to the original training set. This method, which leverages intrinsic batch normalization layers' statistics of the trained model, can be used to evaluate data similarity. Our approach opens a path towards genuine data-free model compression, alleviating the need for training data during model deployment.