mlsanityData to 14 Oct 2026

Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo

arXiv 2203.17248 ↗First posted 30 Mar 2022cs.LG cs.AIcited by 52

Not on the map. Why?

Abstract (arXiv)

Contrastive learning (CL) is widely known to require many negative samples, 65536 in MoCo for instance, for which the performance of a dictionary-free framework is often inferior because the negative sample size (NSS) is limited by its mini-batch size (MBS). To decouple the NSS from the MBS, a dynamic dictionary has been adopted in a large volume of CL frameworks, among which arguably the most popular one is MoCo family. In essence, MoCo adopts a momentum-based queue dictionary, for which we perform a fine-grained analysis of its size and consistency. We point out that InfoNCE loss used in MoCo implicitly attract anchors to their corresponding positive sample with various strength of penalties and identify such inter-anchor hardness-awareness property as a major reason for the necessity of a large dictionary. Our findings motivate us to simplify MoCo v2 via the removal of its dictionary as well as momentum. Based on an InfoNCE with the proposed dual temperature, our simplified frameworks, SimMoCo and SimCo, outperform MoCo v2 by a visible margin. Moreover, our work bridges the gap between CL and non-CL frameworks, contributing to a more unified understanding of these two mainstream frameworks in SSL. Code is available at: https://bit.ly/3LkQbaT.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 54 of its 64 references that are in the map. First posted on arXiv 30 Mar 2022.

Builds on

54 of 64 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  10. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
38 more
  1. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  5. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  6. 2019Contrastive Multiview Codingcited by 1,301
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2017In Defense of the Triplet Loss for Person Re-Identificationcited by 786
  11. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  12. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  13. 2020Contrastive Learning with Hard Negative Samplescited by 628
  14. 2017Large Batch Training of Convolutional Networkscited by 535
  15. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  16. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  17. 2020Understanding the Behaviour of Contrastive Losscited by 398
  18. 2020Debiased Contrastive Learningcited by 356
  19. 2020Whitening for Self-Supervised Representation Learningcited by 279
  20. 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
  21. 2021Contrastive Learning for Compact Single Image Dehazingcited by 219
  22. 2021DetCo: Unsupervised Contrastive Learning for Object Detectioncited by 205
  23. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  24. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 191
  25. 2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
  26. 2021Decoupled Contrastive Learningcited by 143
  27. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  28. 2020Contrastive Learning with Adversarial Examplescited by 91
  29. 2019Hierarchical Clustering with Hard-batch Triplet Loss for Person Re-identificationcited by 83
  30. 2021Jo-SRC: A Contrastive Approach for Combating Noisy Labelscited by 80
  31. 2021Group-aware Contrastive Regression for Action Quality Assessmentcited by 75
  32. 2020Sequence-to-Sequence Contrastive Learning for Text Recognitioncited by 70
  33. 2020Conditional Negative Sampling for Contrastive Learning of Visual Representationscited by 57
  34. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  35. 2021Instance-wise Hard Negative Example Generation for Contrastive Learning in Unpaired Image-to-Image Translationcited by 45
  36. 2021Improving Contrastive Learning by Visualizing Feature Transformationcited by 44
  37. 2019Learning to Discriminate Information for Online Action Detectioncited by 37
  38. 2018Mining on Manifolds: Metric Learning without Labelscited by 36

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 18; 2024: 13; 2025: 10; 2026: 4.

52 papers.

Sort:

By area:

  1. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 352
  2. 2023A Cookbook of Self-Supervised Learningcited by 250
  3. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  4. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  5. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  6. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  7. 2023Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Datacited by 44
  8. 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 43
  9. 2023MobileSAMv2: Faster Segment Anything to Everythingcited by 42
  10. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  11. 2023Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualizationcited by 25
  12. 2022Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustnesscited by 20
38 more
  1. 2023MP-FedCL: Multiprototype Federated Contrastive Learning for Edge Intelligencecited by 20
  2. 2023DomainAdaptor: A Novel Approach to Test-time Adaptationcited by 20
  3. 2024Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervisioncited by 13
  4. 2024Cross-view Masked Diffusion Transformers for Person Image Synthesiscited by 12
  5. 2022Model-Aware Contrastive Learning: Towards Escaping the Dilemmascited by 11
  6. 2022On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learningcited by 10
  7. 2023A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learningcited by 10
  8. 2025FreCT: Frequency-augmented Convolutional Transformer for Robust Time Series Anomaly Detectioncited by 10
  9. 2024FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resourcescited by 9
  10. 2024Mobility-Aware Federated Self-supervised Learning in Vehicular Networkcited by 9
  11. 2024Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?cited by 8
  12. 2022Self-Supervised Visual Representation Learning via Residual Momentumcited by 7
  13. 2024Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Lossescited by 7
  14. 2024DRL-Based Resource Allocation for Motion Blur Resistant Federated Self-Supervised Learning in IoVcited by 7
  15. 2024Analysis of Using Sigmoid Loss for Contrastive Learningcited by 6
  16. 2024SCE-MAE: Selective Correspondence Enhancement with Masked Autoencoder for Self-Supervised Landmark Estimationcited by 6
  17. 2026TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddingscited by 6
  18. 2023BSL: Understanding and Improving Softmax Loss for Recommendationcited by 5
  19. 2024Lite-SAM Is Actually What You Need for Segment Everythingcited by 5
  20. 2024SynCo: Synthetic Hard Negatives for Contrastive Visual Representation Learningcited by 5
  21. 2023Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self-Supervision, Reconstruction, Projector Headcited by 4
  22. 2023SAM Meets UAP: Attacking Segment Anything Model With Universal Adversarial Perturbationcited by 4
  23. 2025A Principled Framework for Multi-View Contrastive Learningcited by 4
  24. 2023INTapt: Information-Theoretic Adversarial Prompt Tuning for Enhanced Non-Native Speech Recognitioncited by 3
  25. 2024Self-Supervised Learning for Text Recognition: A Critical Surveycited by 3
  26. 2023M33D: Learning 3D priors using Multi-Modal Masked Autoencoders for 2D image and video understandingcited by 2
  27. 2025E-MD3C: Taming Masked Diffusion Transformers for Efficient Zero-Shot Object Customizationcited by 2
  28. 2025Decoupled Doubly Contrastive Learning for Cross Domain Facial Action Unit Detectioncited by 2
  29. 2025How PARTs assemble into wholes: Learning the relative composition of imagescited by 2
  30. 2023Rethinking Samples Selection for Contrastive Learning: Mining of Potential Samplescited by 1
  31. 2025FedSKC: Federated Learning with Non-IID Data via Structural Knowledge Collaborationcited by 1
  32. 2026MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Datacited by 1
  33. 2023Concept-wise Fine-tuning Matters in Preventing Negative Transfercited by 0
  34. 2024A Two-Stage Progressive Pre-training using Multi-Modal Contrastive Masked Autoencoderscited by 0
  35. 2025Boosting Open Set Recognition Performance through Modulated Representation Learningcited by 0
  36. 2025Divergence-Based Similarity Function for Multi-View Contrastive Learningcited by 0
  37. 2025Understanding Self-supervised Contrastive Learning through Supervised Objectivescited by 0
  38. 2025Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generationcited by 0

50 most cited of 52 All 52 (JSON)