mlsanityData to 14 Oct 2026

Contrastive Model Inversion for Data-Free Knowledge Distillation

arXiv 2105.08584 ↗First posted 18 May 2021cs.AI cs.CV cs.LGcited by 66

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Model inversion, whose goal is to recover training data from a pre-trained model, has been recently proved feasible. However, existing inversion methods usually suffer from the mode collapse problem, where the synthesized instances are highly similar to each other and thus show limited effectiveness for downstream tasks, such as knowledge distillation. In this paper, we propose Contrastive Model Inversion~(CMI), where the data diversity is explicitly modeled as an optimizable objective, to alleviate the mode collapse issue. Our main observation is that, under the constraint of the same amount of data, higher data diversity usually indicates stronger instance discrimination. To this end, we introduce in CMI a contrastive learning objective that encourages the synthesizing instances to be distinguishable from the already synthesized ones in previous batches. Experiments of pre-trained models on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CMI not only generates more visually plausible instances than the state of the arts, but also achieves significantly superior performance when the generated data are used for knowledge distillation. Code is available at urlhttps://github.com/zju-vipa/DataFree.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 24 of its 35 references that are in the map. First posted on arXiv 18 May 2021.

Builds on

24 of 35 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2016Wide Residual Networkscited by 3,070
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  9. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  10. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  11. 2019The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networkscited by 246
  12. 2019Data-Free Learning of Student Networkscited by 204
12 more
  1. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  2. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  3. 2020Distilling Knowledge from Graph Convolutional Networkscited by 129
  4. 2020Overcoming Catastrophic Forgetting in Graph Neural Networkscited by 95
  5. 2019Data-Free Adversarial Distillationcited by 90
  6. 2020Data-Free Network Quantization With Adversarial Knowledge Distillationcited by 80
  7. 2020Factorizable Graph Convolutional Networkscited by 79
  8. 2018Amalgamating Knowledge towards Comprehensive Classificationcited by 38
  9. 2020Data-Free Knowledge Amalgamation via Group-Stack Dual-GANcited by 32
  10. 2019Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and Morecited by 29
  11. 2020Large-Scale Generative Data-Free Distillationcited by 29
  12. 2020Adversarial Self-Supervised Data-Free Distillation for Text Classificationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 5; 2023: 21; 2024: 20; 2025: 10; 2026: 7.

66 papers.

Sort:

By area:

  1. 2022Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replaycited by 35
  2. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  3. 2022Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillationcited by 31
  4. 2024Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defensescited by 30
  5. 2023Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillationcited by 29
  6. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 26
  7. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 24
  8. 2023NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillationcited by 22
  9. 2024DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecturecited by 22
  10. 2023Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?cited by 21
  11. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  12. 2024Small Scale Data-Free Knowledge Distillationcited by 20
38 more
  1. 2023Learning to Learn from APIs: Black-Box Data-Free Meta-Learningcited by 18
  2. 2022IDEAL: Query-Efficient Data-Free Learning from Black-box Modelscited by 16
  3. 2023On Strengthening and Defending Graph Reconstruction Attack with Markov Chain Approximationcited by 16
  4. 2024Text-Enhanced Data-free Approach for Federated Class-Incremental Learningcited by 16
  5. 2025Data-free Knowledge Distillation with Diffusion Modelscited by 16
  6. 2022CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharingcited by 15
  7. 2024De-confounded Data-free Knowledge Distillation for Handling Distribution Shiftscited by 11
  8. 2023Architecture, Dataset and Model-Scale Agnostic Data-free Meta-Learningcited by 10
  9. 2023FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image Recognitioncited by 10
  10. 2023Revisiting Data-Free Knowledge Distillation with Poisoned Teacherscited by 9
  11. 2024What Makes a Good Dataset for Knowledge Distillation?cited by 9
  12. 2023Explicit and Implicit Knowledge Distillation via Unlabeled Datacited by 8
  13. 2023Distribution Shift Matters for Knowledge Distillation with Webly Collected Imagescited by 8
  14. 2023Sampling to Distill: Knowledge Transfer from Open-World Datacited by 8
  15. 2022Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning Strategycited by 7
  16. 2023Data-Free Sketch-Based Image Retrievalcited by 7
  17. 2023Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillationcited by 7
  18. 2023Data-Free Hard-Label Robustness Stealing Attackcited by 6
  19. 2024Relation-Guided Adversarial Learning for Data-free Knowledge Transfercited by 6
  20. 2024Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversificationcited by 5
  21. 2025When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Needcited by 5
  22. 2023Task-Distributionally Robust Data-Free Meta-Learningcited by 4
  23. 2024Data-free Knowledge Distillation for Fine-grained Visual Categorizationcited by 4
  24. 2024Efficient and Effective Model Extractioncited by 4
  25. 2024Large-Scale Data-Free Knowledge Distillation for ImageNet via Multi-Resolution Data Generationcited by 4
  26. 2024Toward Efficient Data-Free Unlearningcited by 4
  27. 2025Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applicationscited by 4
  28. 2024Encapsulating Knowledge in One Promptcited by 3
  29. 2024Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformerscited by 3
  30. 2021Up to 100× Faster Data-free Knowledge Distillationcited by 2
  31. 2023Customizing Synthetic Data for Data-Free Student Learningcited by 2
  32. 2023Towards Few-Call Model Stealing via Active Self-Paced Knowledge Distillation and Diffusion-Based Image Generationcited by 2
  33. 2023Data-Free Knowledge Distillation Using Adversarially Perturbed OpenGL Shader Imagescited by 2
  34. 2024DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learningcited by 2
  35. 2025CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillationcited by 2
  36. 2023Feature-Rich Audio Model Inversion for Data-Free Knowledge Distillation Towards General Sound Classificationcited by 1
  37. 2024Optimizing Vision Transformers with Data-Free Knowledge Transfercited by 1
  38. 2024Condensed Data Expansion Using Model Inversion for Knowledge Distillationcited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models
  2. 2024Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration
  3. 2024Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation
  4. 2024iDAT: inverse Distillation Adapter-Tuning
  5. 2023Deep Classifier Mimicry without Data Access
  6. 2024Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architectures
  7. 2025Flipping Knowledge Distillation: Leveraging Small Models' Expertise to Enhance LLMs in Text Matching
  8. 2024Discriminative and Consistent Representation Distillation