mlsanityData to 14 Oct 2026

Deep Classifier Mimicry without Data Access

arXiv 2306.02090 ↗First posted 3 Jun 2023cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Access to pre-trained models has recently emerged as a standard across numerous machine learning domains. Unfortunately, access to the original data the models were trained on may not equally be granted. This makes it tremendously challenging to fine-tune, compress models, adapt continually, or to do any other type of data-driven update. We posit that original data access may however not be required. Specifically, we propose Contrastive Abductive Knowledge Extraction (CAKE), a model-agnostic knowledge distillation procedure that mimics deep classifiers without access to the original data. To this end, CAKE generates pairs of noisy synthetic samples and diffuses them contrastively toward a model's decision boundary. We empirically corroborate CAKE's effectiveness using several benchmark datasets and various architectural choices, paving the way for broad application.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 37 of its 62 references that are in the map. First posted on arXiv 3 Jun 2023.

Builds on

37 of 62 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  8. 2016Wide Residual Networkscited by 3,070
  9. 2016Learning without Forgettingcited by 2,496
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  11. 2020Knowledge Distillation: A Surveycited by 1,603
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
25 more
  1. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  2. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  3. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  4. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  5. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  6. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  7. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  8. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  9. 2022Cold Diffusion: Inverting Arbitrary Image Transforms Without Noisecited by 276
  10. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 266
  11. 2019Data-Free Learning of Student Networkscited by 204
  12. 2019Learning Lightweight Lane Detection CNNs by Self Attention Distillationcited by 162
  13. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  14. 2021Vision Transformer for Small-Size Datasetscited by 122
  15. 2019Zero-Shot Knowledge Distillation in Deep Networkscited by 121
  16. 2022Soft Diffusion: Score Matching for General Corruptionscited by 102
  17. 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
  18. 2019Analyzing and Improving Representations with the Soft Nearest Neighbor Losscited by 95
  19. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
  20. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  21. 2021Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesiscited by 14
  22. 2016Knowledge Distillation for Small-footprint Highway Networkscited by 13
  23. 2020Robustness and Diversity Seeking Data-Free Knowledge Distillationcited by 9
  24. 2015Distilling Word Embeddings: An Encoding Approachcited by 7
  25. 2018Low-Dimensional Bottleneck Features for On-Device Continuous Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Tractable Representation Learning with Probabilistic Circuitscited by 4
  2. 2025Single Domain Generalization with Model-aware Parametric Batch-wise Mixupcited by 0
  3. 2025Single Domain Generalization with Adversarial Memorycited by 0
  4. 2025Task-conditioned Ensemble of Expert Models for Continuous Learningcited by 0
  5. 2026Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTscited by 0
  6. 2026On the Failure of Boundary-Seeking Distillation in Bottlenecked Generative Architecturescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Contrastive Model Inversion for Data-Free Knowledge Distillation
  2. 2024Discriminative and Consistent Representation Distillation
  3. 2024Contrastive Unlearning: A Contrastive Approach to Machine Unlearning
  4. 2026BicKD: Bilateral Contrastive Knowledge Distillation
  5. 2021Mining Data Impressions from Deep Models as Substitute for the Unavailable Training Data
  6. 2024Improve Cross-Architecture Generalization on Dataset Distillation
  7. 2024On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models
  8. 2023Explicit and Implicit Knowledge Distillation via Unlabeled Data