# Dream Distillation: A Data-Independent Model Compression Framework

arXiv: https://arxiv.org/abs/1905.07072 · HTML: https://mlsanity.com/paper/1905.07072/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2019.

## Built on by (most cited first, 19 of 19)

- [DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers](https://mlsanity.com/paper/2204.12997/) (2022)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning](https://mlsanity.com/paper/2305.13888/) (2023)
- [VkD: Improving Knowledge Distillation using Orthogonal Projections](https://mlsanity.com/paper/2403.06213/) (2024)
- [Small Scale Data-Free Knowledge Distillation](https://mlsanity.com/paper/2406.07876/) (2024)
- [IDEAL: Query-Efficient Data-Free Learning from Black-box Models](https://mlsanity.com/paper/2205.11158/) (2022)
- [Generative Zero-shot Network Quantization](https://mlsanity.com/paper/2101.08430/) (2021)
- [De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts](https://mlsanity.com/paper/2403.19539/) (2024)
- [Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2104.00875/) (2021)
- [Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation](https://mlsanity.com/paper/2102.11638/) (2021)
- [Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation](https://mlsanity.com/paper/2303.11611/) (2023)
- [Relation-Guided Adversarial Learning for Data-free Knowledge Transfer](https://mlsanity.com/paper/2412.11380/) (2024)
- [Representation Consolidation for Training Expert Students](https://mlsanity.com/paper/2107.08039/) (2021)
- [Scalable Collaborative Learning via Representation Sharing](https://mlsanity.com/paper/2211.10943/) (2022)
- [Super-Efficient Super Resolution for Fast Adversarial Defense at the Edge](https://mlsanity.com/paper/2112.14340/) (2021)
- [Data-free Distillation with Degradation-prompt Diffusion for Multi-weather Image Restoration](https://mlsanity.com/paper/2409.03455/) (2024)
- [Explaining Neural Networks without Access to Training Data](https://mlsanity.com/paper/2206.04891/) (2022)
- [From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs](https://mlsanity.com/paper/2504.13471/) (2025)
- [Learning to Project for Cross-Task Knowledge Distillation](https://mlsanity.com/paper/2403.14494/) (2024)

## Abstract (arXiv)

Model compression is eminently suited for deploying deep learning on IoT-devices. However, existing model compression techniques rely on access to the original or some alternate dataset. In this paper, we address the model compression problem when no real data is available, e.g., when data is private. To this end, we propose Dream Distillation, a data-independent model compression framework. Our experiments show that Dream Distillation can achieve 88.5% accuracy on the CIFAR-10 test set without actually training on the original data!