mlsanityData to 7 Oct 2026

A Unified Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability

arXiv 2404.02462 ↗First posted 3 Apr 2024cs.CVcited by 9

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Self-supervised learning shows promise in harnessing extensive unlabeled data, but it also confronts significant privacy concerns, especially in vision. In this paper, we aim to perform membership inference on visual self-supervised models in a more realistic setting: self-supervised training method and details are unknown for an adversary when attacking as he usually faces a black-box system in practice. In this setting, considering that self-supervised model could be trained by completely different self-supervised paradigms, e.g., masked image modeling and contrastive learning, with complex training details, we propose a unified membership inference method called PartCrop. It is motivated by the shared part-aware capability among models and stronger part response on the training data. Specifically, PartCrop crops parts of objects in an image to query responses with the image in representation space. We conduct extensive attacks on self-supervised models with different training protocols and structures using three widely used image datasets. The results verify the effectiveness and generalization of PartCrop. Moreover, to defend against PartCrop, we evaluate two common approaches, i.e., early stop and differential privacy, and propose a tailored method called shrinking crop scale range. The defense experiments indicate that all of them are effective. Our code is available at https://github.com/JiePKU/PartCrop.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 56 of its 92 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

56 of 92 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022M^4I: Multi-modal Models Membership Inference15 Sep
  2. 2021Membership Inference Attacks Against Self-supervised Speech Models9 Nov
  3. 2021Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture15 Oct
  4. 2021EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learning25 Aug
  5. 2021Membership Inference on Word Embedding and Beyond21 Jun
  6. 2021Membership Inference Attacks on Machine Learning: A Survey14 Mar
  7. 2021ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models4 Feb

Other references · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2023Segment Anythingcited by 8,158
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2022Generative Adversarial Networkscited by 5,603
37 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  10. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  12. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  13. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  14. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
  15. 2020Label-Only Membership Inference Attackscited by 371
  16. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 273
  17. 2020Information Leakage in Embedding Modelscited by 199
  18. 2019MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examplescited by 183
  19. 2018Machine Learning with Membership Privacy using Adversarial Regularizationcited by 177
  20. 2020Adversarial Self-Supervised Contrastive Learningcited by 138
  21. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  22. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  23. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  24. 2021Backdoor Attacks on Self-Supervised Learningcited by 89
  25. 2020Quantifying Privacy Leakage in Graph Embeddingcited by 82
  26. 2020MixCo: Mix-up Contrastive Learning for Visual Representationcited by 62
  27. 2022Understanding Masked Image Modeling via Learning Occlusion Invariant Featurecited by 50
  28. 2022SSLGuard: A Watermarking Scheme for Self-supervised Learning Pre-trained Encoderscited by 43
  29. 2022Dataset Inference for Self-Supervised Modelscited by 36
  30. 2022PoisonedEncoder: Poisoning the Unlabeled Pre-training Data in Contrastive Learningcited by 33
  31. 2020Against Membership Inference Attack: Pruning is All You Needcited by 32
  32. 2022StolenEncoder: Stealing Pre-trained Encoders in Self-supervised Learningcited by 21
  33. 2022Is Self-Supervised Learning More Robust Than Supervised Learning?cited by 17
  34. 2022Exploring Stochastic Autoregressive Image Modeling for Visual Representationcited by 14
  35. 2022Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software Deploymentcited by 8
  36. 2021E-CRF: Embedded Conditional Random Field for Boundary-caused Class Weights Confusion in Semantic Segmentationcited by 7
  37. 2023Understanding Self-Supervised Pretraining with Part-Aware Representation Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 1.

9 papers.

  1. 2024MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Expertscited by 15
  2. 2025Instance-Level Data-Use Auditing of Visual ML Modelscited by 4
  3. 2025When Better Features Mean Greater Risks: The Performance-Privacy Trade-Off in Contrastive Learningcited by 3
  4. 2025CompLeak: Deep Learning Model Compression Exacerbates Privacy Leakagecited by 3
  5. 2025Auditing Data Provenance in Real-world Text-to-Image Diffusion Models for Privacy and Copyright Protectioncited by 2
  6. 2025Dataset Ownership Verification for Pre-trained Masked Modelscited by 2
  7. 2025A Unified and Scalable Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capabilitycited by 1
  8. 2024Pre-trained Encoder Inference: Revealing Upstream Encoders In Downstream Machine Learning Servicescited by 0
  9. 2026When T2I Synthetic Data Backfires: Amplified Privacy Risks in Real-Synthetic Mix Trainingcited by 0