mlsanityData to 14 Oct 2026

The Knowledge Within: Methods for Data-Free Model Compression

arXiv 1912.01274 ↗First posted 3 Dec 2019cs.LG cs.CV stat.MLcited by 54

Not on the map. Why?

Abstract (arXiv)

Recently, an extensive amount of research has been focused on compressing and accelerating Deep Neural Networks (DNN). So far, high compression rate algorithms require part of the training dataset for a low precision calibration, or a fine-tuning process. However, this requirement is unacceptable when the data is unavailable or contains sensitive information, as in medical and biometric use-cases. We present three methods for generating synthetic samples from trained models. Then, we demonstrate how these samples can be used to calibrate and fine-tune quantized models without using any real data in the process. Our best performing method has a negligible accuracy degradation compared to the original training set. This method, which leverages intrinsic batch normalization layers' statistics of the trained model, can be used to evaluate data similarity. Our approach opens a path towards genuine data-free model compression, alleviating the need for training data during model deployment.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 26; 2022: 13; 2023: 6; 2024: 6; 2025: 1; 2026: 2.

54 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  2. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
  3. 2021Source-Free Domain Adaptation for Semantic Segmentationcited by 168
  4. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 131
  5. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  6. 2021Diversifying Sample Generation for Accurate Data-Free Quantizationcited by 63
  7. 2021IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantizationcited by 43
  8. 2021Few-shot Backdoor Defense Using Shapley Estimationcited by 41
  9. 2023A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Taskscited by 41
  10. 2022Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replaycited by 35
  11. 2021Visualizing Adapted Knowledge in Domain Transfercited by 32
  12. 2021Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Datacited by 32
38 more
  1. 2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
  2. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  3. 2023Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?cited by 21
  4. 2022It's All In the Teacher: Zero-Shot Quantization Brought Closer to the Teachercited by 20
  5. 2021Integer-only Zero-shot Quantization for Efficient Speech Recognitioncited by 19
  6. 2024Differentially Private Knowledge Distillation via Synthetic Text Generationcited by 18
  7. 2021AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compressioncited by 17
  8. 2022Genie: Show Me the Data for Quantizationcited by 17
  9. 2022IDEAL: Query-Efficient Data-Free Learning from Black-box Modelscited by 16
  10. 2021Generative Zero-shot Network Quantizationcited by 15
  11. 2021PrivateSNN: Privacy-Preserving Spiking Neural Networkscited by 15
  12. 2022CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharingcited by 15
  13. 2023Hard Sample Matters a Lot in Zero-Shot Quantizationcited by 15
  14. 2024ZipNN: Lossless Compression for AI Modelscited by 14
  15. 2024Self-calibration for Language Model Quantization and Pruningcited by 13
  16. 2021Diverse Sample Generation: Pushing the Limit of Generative Data-free Quantizationcited by 11
  17. 2021Dual Discriminator Adversarial Distillation for Data-free Model Compressioncited by 9
  18. 2021Fine-grained Data Distribution Alignment for Post-Training Quantizationcited by 9
  19. 2021Privacy Vulnerability of Split Computing to Data-Free Model Inversion Attackscited by 8
  20. 2022Compressing Models with Few Samples: Mimicking then Replacingcited by 8
  21. 2023GenQ: Quantization in Low Data Regimes with Generative Synthetic Datacited by 8
  22. 2025Zero-shot Quantization: A Comprehensive Surveycited by 8
  23. 2021Mining Data Impressions from Deep Models as Substitute for the Unavailable Training Datacited by 7
  24. 2022Mixed-Precision Neural Networks: A Surveycited by 7
  25. 2023NaturalInversion: Data-Free Image Synthesis Improving Real-World Consistencycited by 7
  26. 2021Neuroevolution-Enhanced Multi-Objective Optimization for Mixed-Precision Quantizationcited by 6
  27. 2021A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statisticscited by 6
  28. 2021Representation Consolidation for Training Expert Studentscited by 5
  29. 2024Lossless and Near-Lossless Compression for Foundation Modelscited by 5
  30. 2023Don't Memorize; Mimic The Past: Federated Class Incremental Learning Without Episodic Memorycited by 4
  31. 2024Overcoming Catastrophic Forgetting in Federated Class-Incremental Learning via Federated Global Twin Generatorcited by 4
  32. 2026Cut Less, Fold More: Model Compression through the Lens of Projection Geometrycited by 4
  33. 2021Is In-Domain Data Really Needed? A Pilot Study on Cross-Domain Calibration for Network Quantizationcited by 2
  34. 2021Beyond Classification: Knowledge Distillation using Multi-Object Impressionscited by 2
  35. 2021TAGPerson: A Target-Aware Generation Pipeline for Person Re-identificationcited by 2
  36. 2024Data Generation for Hardware-Friendly Post-Training Quantizationcited by 2
  37. 2022Synthetic Dataset Generation for Privacy-Preserving Machine Learningcited by 1
  38. 2026SynQ: Accurate Zero-shot Quantization by Synthesis-aware Fine-tuningcited by 1

50 most cited of 54 All 54 (JSON)