# Extraction of Complex DNN Models: Real Threat or Boogeyman?

arXiv: https://arxiv.org/abs/1910.05429 · HTML: https://mlsanity.com/paper/1910.05429/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2019.

## Built on by (most cited first, 19 of 19)

- [I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences](https://mlsanity.com/paper/2206.08451/) (2022)
- [SoK: How Robust is Image Classification Deep Neural Network Watermarking? (Extended Version)](https://mlsanity.com/paper/2108.04974/) (2021)
- [Increasing the Cost of Model Extraction with Calibrated Proof of Work](https://mlsanity.com/paper/2201.09243/) (2022)
- [False Claims against Model Ownership Resolution](https://mlsanity.com/paper/2304.06607/) (2023)
- [GrOVe: Ownership Verification of Graph Neural Networks using Embeddings](https://mlsanity.com/paper/2304.08566/) (2023)
- [On the Effectiveness of Dataset Watermarking in Adversarial Settings](https://mlsanity.com/paper/2202.12506/) (2022)
- [Conflicting Interactions Among Protection Mechanisms for Machine Learning Models](https://mlsanity.com/paper/2207.01991/) (2022)
- [Thief, Beware of What Get You There: Towards Understanding Model Extraction Attack](https://mlsanity.com/paper/2104.05921/) (2021)
- [Stateful Detection of Model Extraction Attacks](https://mlsanity.com/paper/2107.05166/) (2021)
- [On the Robustness of Dataset Inference](https://mlsanity.com/paper/2210.13631/) (2022)
- [MalProtect: Stateful Defense Against Adversarial Query Attacks in ML-based Malware Detection](https://mlsanity.com/paper/2302.10739/) (2023)
- [Careful What You Wish For: on the Extraction of Adversarially Trained Models](https://mlsanity.com/paper/2207.10561/) (2022)
- [Privacy Inference Attacks and Defenses in Cloud-based Deep Neural Network: A Survey](https://mlsanity.com/paper/2105.06300/) (2021)
- [DynaMarks: Defending Against Deep Learning Model Extraction Using Dynamic Watermarking](https://mlsanity.com/paper/2207.13321/) (2022)
- [Good Artists Copy, Great Artists Steal: Model Extraction Attacks Against Image Translation Models](https://mlsanity.com/paper/2104.12623/) (2021)
- [HODA: Hardness-Oriented Detection of Model Extraction Attacks](https://mlsanity.com/paper/2106.11424/) (2021)
- [Threats, Vulnerabilities, and Controls of Machine Learning Based Systems: A Survey and Taxonomy](https://mlsanity.com/paper/2301.07474/) (2023)
- [Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealing](https://mlsanity.com/paper/2401.16055/) (2024)
- [Attackers Can Do Better: Over- and Understated Factors of Model Stealing Attacks](https://mlsanity.com/paper/2503.06188/) (2025)

## Abstract (arXiv)

Recently, machine learning (ML) has introduced advanced solutions to many domains. Since ML models provide business advantage to model owners, protecting intellectual property of ML models has emerged as an important consideration. Confidentiality of ML models can be protected by exposing them to clients only via prediction APIs. However, model extraction attacks can steal the functionality of ML models using the information leaked to clients through the results returned via the API. In this work, we question whether model extraction is a serious threat to complex, real-life ML models. We evaluate the current state-of-the-art model extraction attack (Knockoff nets) against complex models. We reproduce and confirm the results in the original paper. But we also show that the performance of this attack can be limited by several factors, including ML model architecture and the granularity of API response. Furthermore, we introduce a defense based on distinguishing queries used for Knockoff nets from benign queries. Despite the limitations of the Knockoff nets, we show that a more realistic adversary can effectively steal complex ML models and evade known defenses.