mlsanityData to 7 Oct 2026

Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion

arXiv 1912.08795 ↗First posted 18 Dec 2019cs.LG cs.CV stat.MLcited by 419

Not on the map. Why?

Abstract (arXiv)

We introduce DeepInversion, a new method for synthesizing images from the image distribution used to train a deep neural network. We 'invert' a trained network (teacher) to synthesize class-conditional input images starting from random noise, without using any additional information about the training dataset. Keeping the teacher fixed, our method optimizes the input while regularizing the distribution of intermediate feature maps using information stored in the batch normalization layers of the teacher. Further, we improve the diversity of synthesized images using Adaptive DeepInversion, which maximizes the Jensen-Shannon divergence between the teacher and student network logits. The resulting synthesized images from networks trained on the CIFAR-10 and ImageNet datasets demonstrate high fidelity and degree of realism, and help enable a new breed of data-free applications - ones that do not require any real images or labeled data. We demonstrate the applicability of our proposed method to three tasks of immense practical importance -- (i) data-free network pruning, (ii) data-free knowledge transfer, and (iii) data-free continual learning. Code is available at https://github.com/NVlabs/DeepInversion

Key facts. Cited by 419 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 75; 2022: 72; 2023: 94; 2024: 87; 2025: 55; 2026: 36.

419 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  3. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  4. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  5. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  6. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 320
  7. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  8. 2021Unsolved Problems in ML Safetycited by 291
  9. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  10. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  11. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 171
  12. 2021Source-Free Domain Adaptation for Semantic Segmentationcited by 168
38 more
  1. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 136
  2. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 130
  3. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  4. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 124
  5. 2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
  6. 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 115
  7. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 114
  8. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
  9. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  10. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 107
  11. 2023A Comprehensive Survey on Source-free Domain Adaptationcited by 104
  12. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  13. 2022Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectivescited by 98
  14. 2021Domain Impression: A Source Data Free Domain Adaptation Methodcited by 91
  15. 2022Source-Free Unsupervised Domain Adaptation: A Surveycited by 79
  16. 2022Variational Model Inversion Attackscited by 75
  17. 2022What do Vision Transformers Learn? A Visual Explorationcited by 75
  18. 2022Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replaycited by 70
  19. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 69
  20. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  21. 2021Contrastive Model Inversion for Data-Free Knowledge Distillationcited by 66
  22. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformerscited by 65
  23. 2021Towards Fair Federated Learning with Zero-Shot Data Augmentationcited by 64
  24. 2023TARGET: Federated Class-Continual Learning via Exemplar-Free Distillationcited by 64
  25. 2021Diversifying Sample Generation for Accurate Data-Free Quantizationcited by 63
  26. 2022Expanding Small-Scale Datasets with Guided Imaginationcited by 61
  27. 2022R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learningcited by 59
  28. 2022Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attackscited by 58
  29. 2022Towards Data-Free Model Stealing in a Hard Label Settingcited by 57
  30. 2022Patch Similarity Aware Data-Free Quantization for Vision Transformerscited by 55
  31. 2022A Closer Look at Rehearsal-Free Continual Learningcited by 54
  32. 2022GradViT: Gradient Inversion of Vision Transformerscited by 52
  33. 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 52
  34. 2021A Fast Knowledge Distillation Framework for Visual Recognitioncited by 50
  35. 2021Graph-Free Knowledge Distillation for Graph Neural Networkscited by 46
  36. 2024Lightweight Adaptive Feature De-drifting for Compressed Image Classificationcited by 46
  37. 2021Zero-shot Adversarial Quantizationcited by 44
  38. 2022Do Gradient Inversion Attacks Make Federated Learning Unsafe?cited by 44

50 most cited of 419 All 419 (JSON)