# Can't Steal? Cont-Steal! Contrastive Stealing Attacks Against Image Encoders

arXiv: https://arxiv.org/abs/2201.07513 · HTML: https://mlsanity.com/paper/2201.07513/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 68 references that are in the map.
- First posted on arXiv 19 Jan 2022.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring](https://mlsanity.com/paper/1802.04633/) (2018)
- [Knockoff Nets: Stealing Functionality of Black-Box Models](https://mlsanity.com/paper/1812.02766/) (2018)

## Built on by (most cited first, 30 of 31)

- [I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences](https://mlsanity.com/paper/2206.08451/) (2022)
- [Dataset Inference for Self-Supervised Models](https://mlsanity.com/paper/2209.09024/) (2022)
- [Downstream-agnostic Adversarial Examples](https://mlsanity.com/paper/2307.12280/) (2023)
- [Auditing Membership Leakages of Multi-Exit Networks](https://mlsanity.com/paper/2208.11180/) (2022)
- [MEA-Defender: A Robust Watermark against Model Extraction Attack](https://mlsanity.com/paper/2401.15239/) (2024)
- [Watermarking Vision-Language Pre-trained Models for Multi-modal Embedding as a Service](https://mlsanity.com/paper/2311.05863/) (2023)
- [Bucks for Buckets (B4B): Active Defenses Against Stealing Encoders](https://mlsanity.com/paper/2310.08571/) (2023)
- [Beyond Boundaries: A Comprehensive Survey of Transferable Attacks on AI Systems](https://mlsanity.com/paper/2311.11796/) (2023)
- [A Plot is Worth a Thousand Words: Model Information Stealing Attacks via Scientific Plots](https://mlsanity.com/paper/2302.11982/) (2023)
- [Decoder Gradient Shield: Provable and High-Fidelity Prevention of Gradient-Based Box-Free Watermark Removal](https://mlsanity.com/paper/2502.20924/) (2025)
- [SecurityNet: Assessing Machine Learning Vulnerabilities on Public Models](https://mlsanity.com/paper/2310.12665/) (2023)
- [FDINet: Protecting against DNN Model Extraction via Feature Distortion Index](https://mlsanity.com/paper/2306.11338/) (2023)
- [Training Data Attribution: Was Your Model Secretly Trained On Data Created By Mine?](https://mlsanity.com/paper/2409.15781/) (2024)
- [Clone What You Can't Steal: Black-Box LLM Replication via Logit Leakage and Distillation](https://mlsanity.com/paper/2509.00973/) (2025)
- [New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlook](https://mlsanity.com/paper/2411.07691/) (2024)
- [Examining the Threat Landscape: Foundation Models and Model Stealing](https://mlsanity.com/paper/2502.18077/) (2025)
- [ADAGE: Active Defenses Against GNN Extraction](https://mlsanity.com/paper/2503.00065/) (2025)
- [BESA: Boosting Encoder Stealing Attack with Perturbation Recovery](https://mlsanity.com/paper/2506.04556/) (2025)
- [Dataset Ownership Verification for Pre-trained Masked Models](https://mlsanity.com/paper/2507.12022/) (2025)
- [SSL-Auth: An Authentication Framework by Fragile Watermarking for Pre-trained Encoders in Self-supervised Learning](https://mlsanity.com/paper/2308.04673/) (2023)
- [PromptSync: Bridging Domain Gaps in Vision-Language Models through Class-Aware Prototype Alignment and Discrimination](https://mlsanity.com/paper/2404.07520/) (2024)
- [Decoder Gradient Shields: A Family of Provable and High-Fidelity Methods Against Gradient-Based Box-Free Watermark Removal](https://mlsanity.com/paper/2601.11952/) (2026)
- [DREAM: Domain-free Reverse Engineering Attributes of Black-box Model](https://mlsanity.com/paper/2307.10997/) (2023)
- [Refine, Discriminate and Align: Stealing Encoders via Sample-Wise Prototypes and Multi-Relational Extraction](https://mlsanity.com/paper/2312.00855/) (2023)
- [MEAOD: Model Extraction Attack against Object Detectors](https://mlsanity.com/paper/2312.14677/) (2023)
- [Imitating the Functionality of Image-to-Image Models Using a Single Example](https://mlsanity.com/paper/2406.00828/) (2024)
- [Pre-trained Encoder Inference: Revealing Upstream Encoders In Downstream Machine Learning Services](https://mlsanity.com/paper/2408.02814/) (2024)
- [Your Semantic-Independent Watermark is Fragile: A Semantic Perturbation Attack against EaaS Watermark](https://mlsanity.com/paper/2411.09359/) (2024)
- [DREAM: Domain-agnostic Reverse Engineering Attributes of Black-box Model](https://mlsanity.com/paper/2412.05842/) (2024)
- [Rotation, Scale, and Translation Resilient Black-box Fingerprinting for Intellectual Property Protection of EaaS Models](https://mlsanity.com/paper/2510.16706/) (2025)

## Abstract (arXiv)

Self-supervised representation learning techniques have been developing rapidly to make full use of unlabeled images. They encode images into rich features that are oblivious to downstream tasks. Behind their revolutionary representation power, the requirements for dedicated model designs and a massive amount of computation resources expose image encoders to the risks of potential model stealing attacks - a cheap way to mimic the well-trained encoder performance while circumventing the demanding requirements. Yet conventional attacks only target supervised classifiers given their predicted labels and/or posteriors, which leaves the vulnerability of unsupervised encoders unexplored. In this paper, we first instantiate the conventional stealing attacks against encoders and demonstrate their severer vulnerability compared with downstream classifiers. To better leverage the rich representation of encoders, we further propose Cont-Steal, a contrastive-learning-based attack, and validate its improved stealing effectiveness in various experiment settings. As a takeaway, we appeal to our community's attention to the intellectual property protection of representation learning techniques, especially to the defenses against encoder stealing attacks like ours.