# IDEAL: Query-Efficient Data-Free Learning from Black-box Models

arXiv: https://arxiv.org/abs/2205.11158 · HTML: https://mlsanity.com/paper/2205.11158/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 23 May 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Anti-Backdoor Learning: Training Clean Models on Poisoned Data](https://mlsanity.com/paper/2110.11571/) (2021)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [Zero-shot Knowledge Transfer via Adversarial Belief Matching](https://mlsanity.com/paper/1905.09768/) (2019)
- [Data-Free Model Extraction](https://mlsanity.com/paper/2011.14779/) (2020)
- [DENSE: Data-Free One-Shot Federated Learning](https://mlsanity.com/paper/2112.12371/) (2021)
- [MAZE: Data-Free Model Stealing Attack Using Zeroth-Order Gradient Estimation](https://mlsanity.com/paper/2005.03161/) (2020)
- [Generative Low-bitwidth Data Free Quantization](https://mlsanity.com/paper/2003.03603/) (2020)
- [Heterogeneous Knowledge Distillation using Information Flow Modeling](https://mlsanity.com/paper/2005.00727/) (2020)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [DaST: Data-free Substitute Training for Adversarial Attacks](https://mlsanity.com/paper/2003.12703/) (2020)
- [Inter-Region Affinity Distillation for Road Marking Segmentation](https://mlsanity.com/paper/2004.05304/) (2020)
- [The Knowledge Within: Methods for Data-Free Model Compression](https://mlsanity.com/paper/1912.01274/) (2019)
- [More Grounded Image Captioning by Distilling Image-Text Matching Model](https://mlsanity.com/paper/2004.00390/) (2020)
- [Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model](https://mlsanity.com/paper/2106.03310/) (2021)
- [Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN](https://mlsanity.com/paper/2003.09088/) (2020)
- [Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data](https://mlsanity.com/paper/2110.15094/) (2021)
- [DE-RRD: A Knowledge Distillation Framework for Recommender System](https://mlsanity.com/paper/2012.04357/) (2020)
- [Delving into Data: Effectively Substitute Training for Black-box Attack](https://mlsanity.com/paper/2104.12378/) (2021)
- [Bidirectional Distillation for Top-K Recommender System](https://mlsanity.com/paper/2106.02870/) (2021)
- [Dream Distillation: A Data-Independent Model Compression Framework](https://mlsanity.com/paper/1905.07072/) (2019)
- [Student Surpasses Teacher: Imitation Attack for Black-Box NLP APIs](https://mlsanity.com/paper/2108.13873/) (2021)
- [Privileged Graph Distillation for Cold Start Recommendation](https://mlsanity.com/paper/2105.14975/) (2021)

## Built on by (most cited first, 16 of 16)

- [When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions](https://mlsanity.com/paper/2306.15546/) (2023)
- [No One Left Behind: Real-World Federated Class-Incremental Learning](https://mlsanity.com/paper/2302.00903/) (2023)
- [Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher](https://mlsanity.com/paper/2304.01731/) (2023)
- [A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency](https://mlsanity.com/paper/2307.10655/) (2023)
- [Rethinking Data Distillation: Do Not Overlook Calibration](https://mlsanity.com/paper/2307.12463/) (2023)
- [Does Training with Synthetic Data Truly Protect Privacy?](https://mlsanity.com/paper/2502.12976/) (2025)
- [Fully Exploiting Every Real Sample: SuperPixel Sample Gradient Model Stealing](https://mlsanity.com/paper/2406.18540/) (2024)
- [Efficient and Effective Model Extraction](https://mlsanity.com/paper/2409.14122/) (2024)
- [Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks](https://mlsanity.com/paper/2504.17421/) (2025)
- [Layer-adaptive Structured Pruning Guided by Latency](https://mlsanity.com/paper/2305.14403/) (2023)
- [LDPKiT: Superimposing Remote Queries for Privacy-Preserving Distillation](https://mlsanity.com/paper/2405.16361/) (2024)
- [AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspective](https://mlsanity.com/paper/2603.24857/) (2026)
- [CaBaGe: Data-Free Model Extraction using ClAss BAlanced Generator Ensemble](https://mlsanity.com/paper/2409.10643/) (2024)
- [Stabilizing Data-Free Model Extraction](https://mlsanity.com/paper/2509.11159/) (2025)
- [Diverse Image Priors for Black-box Data-free Knowledge Distillation](https://mlsanity.com/paper/2604.25794/) (2026)
- [Improving Diversity in Black-box Few-shot Knowledge Distillation](https://mlsanity.com/paper/2604.25795/) (2026)

## Abstract (arXiv)

Knowledge Distillation (KD) is a typical method for training a lightweight student model with the help of a well-trained teacher model. However, most KD methods require access to either the teacher's training data or model parameters, which is unrealistic. To tackle this problem, recent works study KD under data-free and black-box settings. Nevertheless, these works require a large number of queries to the teacher model, which incurs significant monetary and computational costs. To address these problems, we propose a novel method called query-effIcient Data-free lEarning from blAck-box modeLs (IDEAL), which aims to query-efficiently learn from black-box model APIs to train a good student without any real data. In detail, IDEAL trains the student model in two stages: data generation and model distillation. Note that IDEAL does not require any query in the data generation stage and queries the teacher only once for each sample in the distillation stage. Extensive experiments on various real-world datasets show the effectiveness of the proposed IDEAL. For instance, IDEAL can improve the performance of the best baseline method DFME by 5.83% on CIFAR10 dataset with only 0.02x the query budget of DFME.