mlsanityData to 7 Oct 2026

On the Memorization Properties of Contrastive Learning

arXiv 2107.10143 ↗First posted 21 Jul 2021cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Memorization studies of deep neural networks (DNNs) help to understand what patterns and how do DNNs learn, and motivate improvements to DNN training approaches. In this work, we investigate the memorization properties of SimCLR, a widely used contrastive self-supervised learning approach, and compare them to the memorization of supervised learning and random labels training. We find that both training objects and augmentations may have different complexity in the sense of how SimCLR learns them. Moreover, we show that SimCLR is similar to random labels training in terms of the distribution of training objects complexity.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 14 of its 18 references that are in the map. First posted on arXiv 21 Jul 2021.

Builds on

14 of 18 references.

References · 14

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  7. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  8. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  9. 2016Colorful Image Colorizationcited by 1,113
  10. 2017A Closer Look at Memorization in Deep Networkscited by 936
  11. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  12. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
2 more
  1. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  2. 2018Training behavior of deep neural network in frequency domaincited by 174

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024Memorization in Self-Supervised Learning Improves Downstream Generalizationcited by 18
  2. 2024Localizing Memorization in SSL Vision Encoderscited by 9
  3. 2025Captured by Captions: On Memorization and its Mitigation in CLIP Modelscited by 4
  4. 2026MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learningcited by 0