mlsanityData to 7 Oct 2026

IDEAL: Query-Efficient Data-Free Learning from Black-box Models

arXiv 2205.11158 ↗First posted 23 May 2022cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Knowledge Distillation (KD) is a typical method for training a lightweight student model with the help of a well-trained teacher model. However, most KD methods require access to either the teacher's training data or model parameters, which is unrealistic. To tackle this problem, recent works study KD under data-free and black-box settings. Nevertheless, these works require a large number of queries to the teacher model, which incurs significant monetary and computational costs. To address these problems, we propose a novel method called query-effIcient Data-free lEarning from blAck-box modeLs (IDEAL), which aims to query-efficiently learn from black-box model APIs to train a good student without any real data. In detail, IDEAL trains the student model in two stages: data generation and model distillation. Note that IDEAL does not require any query in the data generation stage and queries the teacher only once for each sample in the distillation stage. Extensive experiments on various real-world datasets show the effectiveness of the proposed IDEAL. For instance, IDEAL can improve the performance of the best baseline method DFME by 5.83% on CIFAR10 dataset with only 0.02x the query budget of DFME.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

31 of 44 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  7. 2020Knowledge Distillation: A Surveycited by 1,578
  8. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  9. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 258
  10. 2019Data-Free Learning of Student Networkscited by 203
  11. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  12. 2020Data-Free Model Extractioncited by 121
19 more
  1. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  2. 2020MAZE: Data-Free Model Stealing Attack Using Zeroth-Order Gradient Estimationcited by 99
  3. 2020Generative Low-bitwidth Data Free Quantizationcited by 87
  4. 2020Heterogeneous Knowledge Distillation using Information Flow Modelingcited by 73
  5. 2021Contrastive Model Inversion for Data-Free Knowledge Distillationcited by 66
  6. 2020DaST: Data-free Substitute Training for Adversarial Attackscited by 60
  7. 2020Inter-Region Affinity Distillation for Road Marking Segmentationcited by 58
  8. 2019The Knowledge Within: Methods for Data-Free Model Compressioncited by 54
  9. 2020More Grounded Image Captioning by Distilling Image-Text Matching Modelcited by 43
  10. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
  11. 2020Data-Free Knowledge Amalgamation via Group-Stack Dual-GANcited by 32
  12. 2021Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Datacited by 31
  13. 2020DE-RRD: A Knowledge Distillation Framework for Recommender Systemcited by 28
  14. 2021Delving into Data: Effectively Substitute Training for Black-box Attackcited by 27
  15. 2021Bidirectional Distillation for Top-K Recommender Systemcited by 21
  16. 2019Dream Distillation: A Data-Independent Model Compression Frameworkcited by 19
  17. 2021Student Surpasses Teacher: Imitation Attack for Black-Box NLP APIscited by 16
  18. 2021Privileged Graph Distillation for Cold Start Recommendationcited by 10
  19. 2020Interpretable Foreground Object Search As Knowledge Distillationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 4; 2025: 3; 2026: 3.

16 papers.

Sort:

By area:

  1. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
  2. 2023No One Left Behind: Real-World Federated Class-Incremental Learningcited by 53
  3. 2023Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teachercited by 25
  4. 2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiencycited by 21
  5. 2023Rethinking Data Distillation: Do Not Overlook Calibrationcited by 13
  6. 2025Does Training with Synthetic Data Truly Protect Privacy?cited by 12
  7. 2024Fully Exploiting Every Real Sample: SuperPixel Sample Gradient Model Stealingcited by 5
  8. 2024Efficient and Effective Model Extractioncited by 4
  9. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  10. 2023Layer-adaptive Structured Pruning Guided by Latencycited by 1
  11. 2024LDPKiT: Superimposing Remote Queries for Privacy-Preserving Distillationcited by 1
  12. 2026AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspectivecited by 1
4 more
  1. 2024CaBaGe: Data-Free Model Extraction using ClAss BAlanced Generator Ensemblecited by 0
  2. 2025Stabilizing Data-Free Model Extractioncited by 0
  3. 2026Diverse Image Priors for Black-box Data-free Knowledge Distillationcited by 0
  4. 2026Improving Diversity in Black-box Few-shot Knowledge Distillationcited by 0