# Copycat CNN: Stealing Knowledge by Persuading Confession with Random Non-Labeled Data

arXiv: https://arxiv.org/abs/1806.05476 · HTML: https://mlsanity.com/paper/1806.05476/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2018.

## Built on by (most cited first, 30 of 70)

- [Dataset Inference: Ownership Resolution in Machine Learning](https://mlsanity.com/paper/2104.10706/) (2021)
- [Proof-of-Learning: Definitions and Practice](https://mlsanity.com/paper/2103.05633/) (2021)
- [Protecting Intellectual Property of Language Generation APIs with Lexical Watermark](https://mlsanity.com/paper/2112.02701/) (2021)
- [I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences](https://mlsanity.com/paper/2206.08451/) (2022)
- [Towards Data-Free Model Stealing in a Hard Label Setting](https://mlsanity.com/paper/2204.11022/) (2022)
- [Copy, Right? A Testing Framework for Copyright Protection of Deep Learning Models](https://mlsanity.com/paper/2112.05588/) (2021)
- [Survey: Leakage and Privacy at Inference Time](https://mlsanity.com/paper/2107.01614/) (2021)
- [DeepSteal: Advanced Model Extractions Leveraging Efficient Weight Stealing in Memories](https://mlsanity.com/paper/2111.04625/) (2021)
- [Distillation-Resistant Watermarking for Model Protection in NLP](https://mlsanity.com/paper/2210.03312/) (2022)
- [Single-Layer Vision Transformers for More Accurate Early Exits with Less Overhead](https://mlsanity.com/paper/2105.09121/) (2021)
- [StolenEncoder: Stealing Pre-trained Encoders in Self-supervised Learning](https://mlsanity.com/paper/2201.05889/) (2022)
- [Increasing the Cost of Model Extraction with Calibrated Proof of Work](https://mlsanity.com/paper/2201.09243/) (2022)
- [Model Extraction Attacks Revisited](https://mlsanity.com/paper/2312.05386/) (2023)
- [False Claims against Model Ownership Resolution](https://mlsanity.com/paper/2304.06607/) (2023)
- [IronForge: An Open, Secure, Fair, Decentralized Federated Learning](https://mlsanity.com/paper/2301.04006/) (2023)
- [On the Effectiveness of Dataset Watermarking in Adversarial Settings](https://mlsanity.com/paper/2202.12506/) (2022)
- [Conflicting Interactions Among Protection Mechanisms for Machine Learning Models](https://mlsanity.com/paper/2207.01991/) (2022)
- [Marich: A Query-efficient Distributionally Equivalent Model Extraction Attack using Public Data](https://mlsanity.com/paper/2302.08466/) (2023)
- [Watermarking Vision-Language Pre-trained Models for Multi-modal Embedding as a Service](https://mlsanity.com/paper/2311.05863/) (2023)
- [Morphence: Moving Target Defense Against Adversarial Examples](https://mlsanity.com/paper/2108.13952/) (2021)
- [Threat Assessment in Machine Learning based Systems](https://mlsanity.com/paper/2207.00091/) (2022)
- [QUEEN: Query Unlearning against Model Extraction](https://mlsanity.com/paper/2407.01251/) (2024)
- [Stateful Detection of Model Extraction Attacks](https://mlsanity.com/paper/2107.05166/) (2021)
- [Robust and IP-Protecting Vertical Federated Learning against Unexpected Quitting of Parties](https://mlsanity.com/paper/2303.18178/) (2023)
- [QuantumLeak: Stealing Quantum Neural Networks from Cloud-based NISQ Machines](https://mlsanity.com/paper/2403.10790/) (2024)
- [Don't Knock! Rowhammer at the Backdoor of DNN Models](https://mlsanity.com/paper/2110.07683/) (2021)
- [Careful What You Wish For: on the Extraction of Adversarially Trained Models](https://mlsanity.com/paper/2207.10561/) (2022)
- [MeaeQ: Mount Model Extraction Attacks with Efficient Queries](https://mlsanity.com/paper/2310.14047/) (2023)
- [Army of Thieves: Enhancing Black-Box Model Extraction via Ensemble based sample selection](https://mlsanity.com/paper/2311.04588/) (2023)
- [SoK: Unintended Interactions among Machine Learning Defenses and Risks](https://mlsanity.com/paper/2312.04542/) (2023)

## Abstract (arXiv)

In the past few years, Convolutional Neural Networks (CNNs) have been achieving state-of-the-art performance on a variety of problems. Many companies employ resources and money to generate these models and provide them as an API, therefore it is in their best interest to protect them, i.e., to avoid that someone else copies them. Recent studies revealed that state-of-the-art CNNs are vulnerable to adversarial examples attacks, and this weakness indicates that CNNs do not need to operate in the problem domain (PD). Therefore, we hypothesize that they also do not need to be trained with examples of the PD in order to operate in it. Given these facts, in this paper, we investigate if a target black-box CNN can be copied by persuading it to confess its knowledge through random non-labeled data. The copy is two-fold: i) the target network is queried with random data and its predictions are used to create a fake dataset with the knowledge of the network; and ii) a copycat network is trained with the fake dataset and should be able to achieve similar performance as the target network. This hypothesis was evaluated locally in three problems (facial expression, object, and crosswalk classification) and against a cloud-based API. In the copy attacks, images from both non-problem domain and PD were used. All copycat networks achieved at least 93.7% of the performance of the original models with non-problem domain data, and at least 98.6% using additional data from the PD. Additionally, the copycat CNN successfully copied at least 97.3% of the performance of the Microsoft Azure Emotion API. Our results show that it is possible to create a copycat CNN by simply querying a target network as black-box with random non-labeled data.