mlsanityData to 7 Oct 2026

EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learning

arXiv 2108.11023 ↗First posted 25 Aug 2021cs.CR cs.CV cs.LGcited by 68

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Given a set of unlabeled images or (image, text) pairs, contrastive learning aims to pre-train an image encoder that can be used as a feature extractor for many downstream tasks. In this work, we propose EncoderMI, the first membership inference method against image encoders pre-trained by contrastive learning. In particular, given an input and a black-box access to an image encoder, EncoderMI aims to infer whether the input is in the training dataset of the image encoder. EncoderMI can be used 1) by a data owner to audit whether its (public) data was used to pre-train an image encoder without its authorization or 2) by an attacker to compromise privacy of the training data when it is private/sensitive. Our EncoderMI exploits the overfitting of the image encoder towards its training data. In particular, an overfitted image encoder is more likely to output more (or less) similar feature vectors for two augmented versions of an input in (or not in) its training dataset. We evaluate EncoderMI on image encoders pre-trained on multiple datasets by ourselves as well as the Contrastive Language-Image Pre-training (CLIP) image encoder, which is pre-trained on 400 million (image, text) pairs collected from the Internet and released by OpenAI. Our results show that EncoderMI can achieve high accuracy, precision, and recall. We also explore a countermeasure against EncoderMI via preventing overfitting through early stopping. Our results show that it achieves trade-offs between accuracy of EncoderMI and utility of the image encoder, i.e., it can reduce the accuracy of EncoderMI, but it also incurs classification accuracy loss of the downstream classifiers built based on the image encoder.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 29 of its 51 references that are in the map. First posted on arXiv 25 Aug 2021.

Builds on

29 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Practical Blind Membership Inference Attack via Differential Comparisons5 Jan

Other references · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2016Deep Learning with Differential Privacycited by 2,682
  9. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2017Deep Setscited by 1,394
  12. 2016Stealing Machine Learning Models via Prediction APIscited by 602
16 more
  1. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
  2. 2020Label-Only Membership Inference Attackscited by 371
  3. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 273
  4. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  5. 2020Information Leakage in Embedding Modelscited by 199
  6. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  7. 2019MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examplescited by 183
  8. 2018Machine Learning with Membership Privacy using Adversarial Regularizationcited by 177
  9. 2020Membership Leakage in Label-Only Exposurescited by 156
  10. 2019Privacy Risks of Securing Machine Learning Models against Adversarial Examplescited by 136
  11. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 120
  12. 2018Stealing Hyperparameters in Machine Learningcited by 115
  13. 2018AttriGuard: A Practical Defense Against Attribute Inference Attacks via Adversarial Machine Learningcited by 58
  14. 2021Quantifying and Mitigating Privacy Risks of Contrastive Learningcited by 35
  15. 2018Differentially Private Data Generative Modelscited by 33
  16. 2020TransMIA: Membership Inference Attacks Using Transfer Shadow Trainingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 14; 2023: 13; 2024: 16; 2025: 15; 2026: 8.

68 papers.

Sort:

By area:

  1. 2022The Privacy Onion Effect: Memorization is Relativecited by 128
  2. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  3. 2022SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learningcited by 47
  4. 2022SSLGuard: A Watermarking Scheme for Self-supervised Learning Pre-trained Encoderscited by 43
  5. 2022Dataset Inference for Self-Supervised Modelscited by 36
  6. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 35
  7. 2021Property Inference Attacks Against GANscited by 32
  8. 2023Downstream-agnostic Adversarial Examplescited by 32
  9. 2024Membership Inference Attacks against Large Vision-Language Modelscited by 31
  10. 2022Can't Steal? Cont-Steal! Contrastive Stealing Attacks Against Image Encoderscited by 30
  11. 2022M^4I: Multi-modal Models Membership Inferencecited by 30
  12. 2023Overconfidence is a Dangerous Thing: Mitigating Membership Inference Attacks by Enforcing Less Confident Predictioncited by 26
38 more
  1. 2022User-Level Membership Inference Attack against Metric Embedding Learningcited by 22
  2. 2022StolenEncoder: Stealing Pre-trained Encoders in Self-supervised Learningcited by 21
  3. 2022Does CLIP Know My Face?cited by 19
  4. 2024Memorization in Self-Supervised Learning Improves Downstream Generalizationcited by 18
  5. 2022Membership Inference Attacks and Generalization: A Causal Perspectivecited by 17
  6. 2024A General Framework for Data-Use Auditing of ML Modelscited by 15
  7. 2023SLMIA-SR: Speaker-Level Membership Inference Attacks against Speaker Recognition Systemscited by 14
  8. 2023TMI! Finetuned Models Leak Private Information from their Pretraining Datacited by 13
  9. 2023MIST: Defending Against Membership Inference Attacks Through Membership-Invariant Subspace Trainingcited by 11
  10. 2022Semi-Leak: Membership Inference Attacks Against Semi-supervised Learningcited by 10
  11. 2024A Unified Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capabilitycited by 9
  12. 2024Localizing Memorization in SSL Vision Encoderscited by 9
  13. 2024TUNI: A Textual Unimodal Detector for Identity Inference in CLIP Modelscited by 8
  14. 2022SecretGen: Privacy Recovery on Pre-Trained Models via Distribution Discriminationcited by 7
  15. 2023Safeguarding Data in Multimodal AI: A Differentially Private Approach to CLIP Trainingcited by 7
  16. 2024DPAdapter: Improving Differentially Private Deep Learning through Noise Tolerance Pre-trainingcited by 7
  17. 2023Beyond Boundaries: A Comprehensive Survey of Transferable Attacks on AI Systemscited by 6
  18. 202110 Security and Privacy Problems in Large Foundation Modelscited by 5
  19. 2022l-Leaks: Membership Inference Attacks with Logitscited by 5
  20. 2022On the Privacy Risks of Cell-Based NAS Architecturescited by 5
  21. 2024Mitigating Privacy Risk in Membership Inference by Convex-Concave Losscited by 5
  22. 2023Defending Our Privacy With Backdoorscited by 4
  23. 2024Safety and Performance, Why Not Both? Bi-Objective Optimized Model Compression against Heterogeneous Attacks Toward AI Software Deploymentcited by 4
  24. 2025GRID: Protecting Training Graph from Link Stealing Attacks on GNN Modelscited by 4
  25. 2025Captured by Captions: On Memorization and its Mitigation in CLIP Modelscited by 4
  26. 2025Instance-Level Data-Use Auditing of Visual ML Modelscited by 4
  27. 2023SoK: Comparing Different Membership Inference Attacks with a Comprehensive Benchmarkcited by 3
  28. 2024Membership Privacy Evaluation in Deep Spiking Neural Networkscited by 3
  29. 2025When Better Features Mean Greater Risks: The Performance-Privacy Trade-Off in Contrastive Learningcited by 3
  30. 2025Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Modelscited by 3
  31. 2025Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probingcited by 3
  32. 2023A Comprehensive Study of Privacy Risks in Curriculum Learningcited by 2
  33. 2024Membership Inference Attack Against Masked Image Modelingcited by 2
  34. 2025Do Fairness Interventions Come at the Cost of Privacy: Evaluations for Binary Classifierscited by 2
  35. 2025How does Bayesian Sampling help Membership Inference Attacks?cited by 2
  36. 2025Measuring D'ej`a vu Memorization Efficientlycited by 2
  37. 2025BESA: Boosting Encoder Stealing Attack with Perturbation Recoverycited by 2
  38. 2024Machine Unlearning in Contrastive Learningcited by 1

50 most cited of 68 All 68 (JSON)