# Dual Student Networks for Data-Free Model Stealing

arXiv: https://arxiv.org/abs/2309.10058 · HTML: https://mlsanity.com/paper/2309.10058/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 18 Sep 2023.

## Builds on (most cited first, 30 of 31)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/1801.00553/) (2018)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Knockoff Nets: Stealing Functionality of Black-Box Models](https://mlsanity.com/paper/1812.02766/) (2018)
- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [Data-Free Model Extraction](https://mlsanity.com/paper/2011.14779/) (2020)
- [MAZE: Data-Free Model Stealing Attack Using Zeroth-Order Gradient Estimation](https://mlsanity.com/paper/2005.03161/) (2020)
- [Data-Free Adversarial Distillation](https://mlsanity.com/paper/1912.11006/) (2019)
- [DaST: Data-free Substitute Training for Adversarial Attacks](https://mlsanity.com/paper/2003.12703/) (2020)
- [Towards Data-Free Model Stealing in a Hard Label Setting](https://mlsanity.com/paper/2204.11022/) (2022)
- [Interpreting Blackbox Models via Model Extraction](https://mlsanity.com/paper/1705.08504/) (2017)
- [Interpretability via Model Extraction](https://mlsanity.com/paper/1706.09773/) (2017)
- [Model Extraction Warning in MLaaS Paradigm](https://mlsanity.com/paper/1711.07221/) (2017)
- [Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model](https://mlsanity.com/paper/2106.03310/) (2021)
- [Black-Box Attacks on Sequential Recommenders via Data-Free Model Extraction](https://mlsanity.com/paper/2109.01165/) (2021)
- [Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay](https://mlsanity.com/paper/2201.03019/) (2022)
- [Black-Box Ripper: Copying black-box models using generative evolutionary algorithms](https://mlsanity.com/paper/2010.11158/) (2020)
- [ES Attack: Model Stealing against Deep Neural Networks without Data Hurdles](https://mlsanity.com/paper/2009.09560/) (2020)
- [Black-Box Dissector: Towards Erasing-based Hard-Label Model Stealing Attack](https://mlsanity.com/paper/2105.00623/) (2021)
- [Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation](https://mlsanity.com/paper/2102.11638/) (2021)
- [Model Extraction and Adversarial Attacks on Neural Networks using Switching Power Information](https://mlsanity.com/paper/2106.08299/) (2021)

## Built on by (most cited first, 17 of 17)

- [Beyond Boundaries: A Comprehensive Survey of Transferable Attacks on AI Systems](https://mlsanity.com/paper/2311.11796/) (2023)
- [Data-Free Hard-Label Robustness Stealing Attack](https://mlsanity.com/paper/2312.05924/) (2023)
- [Fully Exploiting Every Real Sample: SuperPixel Sample Gradient Model Stealing](https://mlsanity.com/paper/2406.18540/) (2024)
- [Stealthy Imitation: Reward-guided Environment-free Policy Stealing](https://mlsanity.com/paper/2405.07004/) (2024)
- [DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learning](https://mlsanity.com/paper/2409.07734/) (2024)
- [Examining the Threat Landscape: Foundation Models and Model Stealing](https://mlsanity.com/paper/2502.18077/) (2025)
- [From Essence to Defense: Adaptive Semantic-aware Watermarking for Embedding-as-a-Service Copyright Protection](https://mlsanity.com/paper/2512.16439/) (2025)
- [VidModEx: Interpretable and Efficient Black Box Model Extraction for High-Dimensional Spaces](https://mlsanity.com/paper/2408.02140/) (2024)
- [Model-Guardian: Protecting against Data-Free Model Stealing Using Gradient Representations and Deceptive Predictions](https://mlsanity.com/paper/2503.18081/) (2025)
- [DREAM: Domain-agnostic Reverse Engineering Attributes of Black-box Model](https://mlsanity.com/paper/2412.05842/) (2024)
- [Attackers Can Do Better: Over- and Understated Factors of Model Stealing Attacks](https://mlsanity.com/paper/2503.06188/) (2025)
- [Stealix: Model Stealing via Prompt Evolution](https://mlsanity.com/paper/2506.05867/) (2025)
- [Assessing Risk of Stealing Proprietary Models for Medical Imaging Tasks](https://mlsanity.com/paper/2506.19464/) (2025)
- [I Stolenly Swear That I Am Up to (No) Good: Design and Evaluation of Model Stealing Attacks](https://mlsanity.com/paper/2508.21654/) (2025)
- [Stabilizing Data-Free Model Extraction](https://mlsanity.com/paper/2509.11159/) (2025)
- [DiMEx: Breaking the Cold Start Barrier in Data-Free Model Extraction via Latent Diffusion Priors](https://mlsanity.com/paper/2601.01688/) (2026)
- [Geometry-Aware Localized Watermarking for Copyright Protection in Embedding-as-a-Service](https://mlsanity.com/paper/2604.11344/) (2026)

## Abstract (arXiv)

Existing data-free model stealing methods use a generator to produce samples in order to train a student model to match the target model outputs. To this end, the two main challenges are estimating gradients of the target model without access to its parameters, and generating a diverse set of training samples that thoroughly explores the input space. We propose a Dual Student method where two students are symmetrically trained in order to provide the generator a criterion to generate samples that the two students disagree on. On one hand, disagreement on a sample implies at least one student has classified the sample incorrectly when compared to the target model. This incentive towards disagreement implicitly encourages the generator to explore more diverse regions of the input space. On the other hand, our method utilizes gradients of student models to indirectly estimate gradients of the target model. We show that this novel training objective for the generator network is equivalent to optimizing a lower bound on the generator's loss if we had access to the target model gradients. We show that our new optimization framework provides more accurate gradient estimation of the target model and better accuracies on benchmark classification datasets. Additionally, our approach balances improved query efficiency with training computation cost. Finally, we demonstrate that our method serves as a better proxy model for transfer-based adversarial attacks than existing data-free model stealing methods.