# MISLEADER: Defending against Model Extraction with Ensembles of Distilled Models

arXiv: https://arxiv.org/abs/2506.02362 · HTML: https://mlsanity.com/paper/2506.02362/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 64 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring](https://mlsanity.com/paper/1802.04633/) (2018)
- [Knockoff Nets: Stealing Functionality of Black-Box Models](https://mlsanity.com/paper/1812.02766/) (2018)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [High Accuracy and High Fidelity Extraction of Neural Networks](https://mlsanity.com/paper/1909.01838/) (2019)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [PRADA: Protecting against DNN Model Stealing Attacks](https://mlsanity.com/paper/1805.02628/) (2018)
- [Data-Free Model Extraction](https://mlsanity.com/paper/2011.14779/) (2020)
- [Dataset Inference: Ownership Resolution in Machine Learning](https://mlsanity.com/paper/2104.10706/) (2021)
- [MAZE: Data-Free Model Stealing Attack Using Zeroth-Order Gradient Estimation](https://mlsanity.com/paper/2005.03161/) (2020)
- [Protecting Intellectual Property of Language Generation APIs with Lexical Watermark](https://mlsanity.com/paper/2112.02701/) (2021)
- [Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks](https://mlsanity.com/paper/1906.10908/) (2019)
- [NSML: Meet the MLaaS platform with a real-world case study](https://mlsanity.com/paper/1810.09957/) (2018)
- [Towards Data-Free Model Stealing in a Hard Label Setting](https://mlsanity.com/paper/2204.11022/) (2022)
- [Defending Against Model Stealing Attacks with Adaptive Misinformation](https://mlsanity.com/paper/1911.07100/) (2019)
- [Model Extraction Warning in MLaaS Paradigm](https://mlsanity.com/paper/1711.07221/) (2017)
- [Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model](https://mlsanity.com/paper/2106.03310/) (2021)
- [MEGEX: Data-Free Model Extraction Attack against Gradient-Based Explainable AI](https://mlsanity.com/paper/2107.08909/) (2021)
- [How to Steer Your Adversary: Targeted and Efficient Model Stealing Defenses with Gradient Redirection](https://mlsanity.com/paper/2206.14157/) (2022)
- [Model Extraction Attacks Revisited](https://mlsanity.com/paper/2312.05386/) (2023)
- [Learning to Learn from APIs: Black-Box Data-Free Meta-Learning](https://mlsanity.com/paper/2305.18413/) (2023)
- [MLModelCI: An Automatic Cloud Platform for Efficient MLaaS](https://mlsanity.com/paper/2006.05096/) (2020)

## Built on by (most cited first, 8 of 8)

- [CREDIT: Certified Ownership Verification of Deep Neural Networks Against Model Extraction Attacks](https://mlsanity.com/paper/2602.20419/) (2026)
- [Intellectual Property in Graph-Based Machine Learning as a Service: Attacks and Defenses](https://mlsanity.com/paper/2508.19641/) (2025)
- [CITED: A Decision Boundary-Aware Signature for GNNs Towards Model Extraction Defense](https://mlsanity.com/paper/2602.20418/) (2026)
- [An Embarrassingly Simple Detector for Model Extraction Attacks in Large Language Model API Traffic](https://mlsanity.com/paper/2606.05725/) (2026)
- [Let Them Steal: Trapping Large Language Model Extraction Attacks with Knowledge Honeypot](https://mlsanity.com/paper/2606.15810/) (2026)
- [LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?](https://mlsanity.com/paper/2605.11301/) (2026)
- [ADS-C: Antidistillation Sampling for Classification](https://mlsanity.com/paper/2607.15467/) (2026)
- [Curvature Cryptanalysis of Smooth Transformer Feed-Forward Networks](https://mlsanity.com/paper/2608.28843/) (2026)

## Abstract (arXiv)

Model extraction attacks aim to replicate the functionality of a black-box model through query access, threatening the intellectual property (IP) of machine-learning-as-a-service (MLaaS) providers. Defending against such attacks is challenging, as it must balance efficiency, robustness, and utility preservation in the real-world scenario. Despite the recent advances, most existing defenses presume that attacker queries have out-of-distribution (OOD) samples, enabling them to detect and disrupt suspicious inputs. However, this assumption is increasingly unreliable, as modern models are trained on diverse datasets and attackers often operate under limited query budgets. As a result, the effectiveness of these defenses is significantly compromised in realistic deployment scenarios. To address this gap, we propose MISLEADER (enseMbles of dIStiLled modEls Against moDel ExtRaction), a novel defense strategy that does not rely on OOD assumptions. MISLEADER formulates model protection as a bilevel optimization problem that simultaneously preserves predictive fidelity on benign inputs and reduces extractability by potential clone models. Our framework combines data augmentation to simulate attacker queries with an ensemble of heterogeneous distilled models to improve robustness and diversity. We further provide a tractable approximation algorithm and derive theoretical error bounds to characterize defense effectiveness. Extensive experiments across various settings validate the utility-preserving and extraction-resistant properties of our proposed defense strategy. Our code is available at https://github.com/LabRAI/MISLEADER.