mlsanityData to 14 Oct 2026

On the Difficulty of Defending Self-Supervised Learning against Model Extraction

arXiv 2205.07890 ↗First posted 16 May 2022cs.LG cs.AI cs.CRcited by 22

Not on the map. Why?

Abstract (arXiv)

Self-Supervised Learning (SSL) is an increasingly popular ML paradigm that trains models to transform complex inputs into representations without relying on explicit labels. These representations encode similarity structures that enable efficient learning of multiple downstream tasks. Recently, ML-as-a-Service providers have commenced offering trained SSL models over inference APIs, which transform user inputs into useful representations for a fee. However, the high cost involved to train these models and their exposure over APIs both make black-box extraction a realistic security threat. We thus explore model stealing attacks against SSL. Unlike traditional model extraction on classifiers that output labels, the victim models here output representations; these representations are of significantly higher dimensionality compared to the low-dimensional prediction scores output by classifiers. We construct several novel attacks and find that approaches that train directly on a victim's stolen representations are query efficient and enable high accuracy for downstream models. We then show that existing defenses against model extraction are inadequate and not easily retrofitted to the specificities of SSL.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 16 May 2022.

Builds on

26 of 32 references.

References · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2020Supervised Contrastive Learningcited by 2,829
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  10. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  11. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  12. 2016Stealing Machine Learning Models via Prediction APIscited by 619
14 more
  1. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  2. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  3. 2018Knockoff Nets: Stealing Functionality of Black-Box Modelscited by 292
  4. 2018Scalable Private Learning with PATEcited by 269
  5. 2019High Accuracy and High Fidelity Extraction of Neural Networkscited by 214
  6. 2018PRADA: Protecting against DNN Model Stealing Attackscited by 165
  7. 2020Entangled Watermarks as a Defense against Model Extractioncited by 149
  8. 2020Data-Free Model Extractioncited by 124
  9. 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 116
  10. 2020The Cost of Training NLP Models: A Concise Overviewcited by 98
  11. 2019Analyzing and Improving Representations with the Soft Nearest Neighbor Losscited by 95
  12. 2021Proof-of-Learning: Definitions and Practicecited by 84
  13. 2019Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attackscited by 81
  14. 2022Increasing the Cost of Model Extraction with Calibrated Proof of Workcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 7; 2024: 4; 2025: 7; 2026: 1.

22 papers.

Sort:

By area:

  1. 2023The False Promise of Imitating Proprietary LLMscited by 263
  2. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  3. 2022I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defencescited by 75
  4. 2022Dataset Inference for Self-Supervised Modelscited by 36
  5. 2023Downstream-agnostic Adversarial Examplescited by 32
  6. 2025Privacy Attacks on Image AutoRegressive Modelscited by 15
  7. 2023Bucks for Buckets (B4B): Active Defenses Against Stealing Encoderscited by 8
  8. 2025Unlocking Post-hoc Dataset Inference with Synthetic Datacited by 7
  9. 2023Beyond Boundaries: A Comprehensive Survey of Transferable Attacks on AI Systemscited by 6
  10. 2023Beyond Labeling Oracles: What does it mean to steal ML models?cited by 4
  11. 2024Efficient Model-Stealing Attacks Against Inductive Graph Neural Networkscited by 3
  12. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
10 more
  1. 2025Examining the Threat Landscape: Foundation Models and Model Stealingcited by 2
  2. 2025ADAGE: Active Defenses Against GNN Extractioncited by 2
  3. 2025BESA: Boosting Encoder Stealing Attack with Perturbation Recoverycited by 2
  4. 2025Membership and Dataset Inference Attacks on Large Audio Generative Modelscited by 2
  5. 2023SSL-Auth: An Authentication Framework by Fragile Watermarking for Pre-trained Encoders in Self-supervised Learningcited by 1
  6. 2025Efficient and Privacy-Preserving Soft Prompt Transfer for LLMscited by 1
  7. 2023Refine, Discriminate and Align: Stealing Encoders via Sample-Wise Prototypes and Multi-Relational Extractioncited by 0
  8. 2024A2-DIDM: Privacy-preserving Accumulator-enabled Auditing for Distributed Identity of DNN Modelcited by 0
  9. 2024Pre-trained Encoder Inference: Revealing Upstream Encoders In Downstream Machine Learning Servicescited by 0
  10. 2026ArmSSL: Adversarial Robust Black-Box Watermarking for Self-Supervised Learning Pre-trained Encoderscited by 0