mlsanityData to 14 Oct 2026

Do SSL Models Have D'ej`a Vu? A Case of Unintended Memorization in Self-supervised Learning

arXiv 2304.13850 ↗First posted 26 Apr 2023cs.CV cs.CR cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) algorithms can produce useful image representations by learning to associate different parts of natural images with one another. However, when taken to the extreme, SSL models can unintendedly memorize specific parts in individual training samples rather than learning semantically meaningful associations. In this work, we perform a systematic study of the unintended memorization of image-specific information in SSL models -- which we refer to as d'ej`a vu memorization. Concretely, we show that given the trained model and a crop of a training image containing only the background (e.g., water, sky, grass), it is possible to infer the foreground object with high accuracy or even visually reconstruct it. Furthermore, we show that d'ej`a vu memorization is common to different SSL algorithms, is exacerbated by certain design choices, and cannot be detected by conventional techniques for evaluating representation quality. Our study of d'ej`a vu memorization reveals previously unknown privacy risks in SSL models, as well as suggests potential practical mitigation strategies. Code is available at https://github.com/facebookresearch/DejaVu.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 26 Apr 2023.

Builds on

34 of 48 references.

References · 34

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  9. 2016Deep Learning with Differential Privacycited by 2,728
  10. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  11. 2020Extracting Training Data from Large Language Modelscited by 2,189
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
22 more
  1. 2017Deep Image Priorcited by 1,098
  2. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  4. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  5. 2023Extracting Training Data from Diffusion Modelscited by 732
  6. 2018Generative Image Inpainting with Contextual Attentioncited by 580
  7. 2017Large Batch Training of Convolutional Networkscited by 535
  8. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  9. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 498
  10. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  11. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  12. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  13. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 271
  14. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 248
  15. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  16. 2022Reconstructing Training Data with Informed Adversariescited by 148
  17. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 147
  18. 2021High Fidelity Visualization of What Your Self-Supervised Representation Knows Aboutcited by 66
  19. 2022Are Attribute Inference Attacks Just Imputation?cited by 49
  20. 2022Bounding Training Data Reconstruction in Private (Deep) Learningcited by 46
  21. 2022Are Your Sensitive Attributes Private? Novel Model Inversion Attribute Inference Attacks on Classification Modelscited by 43
  22. 2023Towards Democratizing Joint-Embedding Self-Supervised Learningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 3; 2026: 2.

19 papers.

Sort:

By area:

  1. 2023Virchow: A Million-Slide Digital Pathology Foundation Modelcited by 90
  2. 2024Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Predictioncited by 41
  3. 2024Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Modelscited by 37
  4. 2024Memorization in Self-Supervised Learning Improves Downstream Generalizationcited by 18
  5. 2023ViP: A Differentially Private Foundation Model for Computer Visioncited by 17
  6. 2024Certified Machine Unlearning via Noisy Stochastic Gradient Descentcited by 14
  7. 2024D'ej`a Vu Memorization in Vision-Language Modelscited by 9
  8. 2024Localizing Memorization in SSL Vision Encoderscited by 9
  9. 2024Extracting Training Data from Document-Based VQA Modelscited by 6
  10. 2025Captured by Captions: On Memorization and its Mitigation in CLIP Modelscited by 4
  11. 2024Noise Masking Attacks and Defenses for Pretrained Speech Modelscited by 3
  12. 2024Views Can Be Deceiving: Improved SSL Through Feature Space Augmentationcited by 2
7 more
  1. 2025Measuring D'ej`a vu Memorization Efficientlycited by 2
  2. 2024Better Membership Inference Privacy Measurement through Discrepancycited by 1
  3. 2024LaFAM: Unsupervised Feature Attribution with Label-free Activation Mapscited by 1
  4. 2024Mitigating Downstream Model Risks via Model Provenancecited by 1
  5. 2025The Impact of Semantic Pairs on Self-Supervised Representation Learningcited by 0
  6. 2026Dataset Watermarking with Provable Black-Box Detectioncited by 0
  7. 2026MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learningcited by 0