mlsanityData to 14 Oct 2026

Dynamically Scaled Temperature in Self-Supervised Contrastive Learning

arXiv 2308.01140 ↗First posted 2 Aug 2023cs.LG cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

In contemporary self-supervised contrastive algorithms like SimCLR, MoCo, etc., the task of balancing attraction between two semantically similar samples and repulsion between two samples of different classes is primarily affected by the presence of hard negative samples. While the InfoNCE loss has been shown to impose penalties based on hardness, the temperature hyper-parameter is the key to regulating the penalties and the trade-off between uniformity and tolerance. In this work, we focus our attention on improving the performance of InfoNCE loss in self-supervised learning by proposing a novel cosine similarity dependent temperature scaling function to effectively optimize the distribution of the samples in the feature space. We also provide mathematical analyses to support the construction of such a dynamically scaled temperature function. Experimental evidence shows that the proposed framework outperforms the contrastive loss-based SSL algorithms.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 2 Aug 2023.

Builds on

43 of 59 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2022Generative Adversarial Networkscited by 5,627
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
31 more
  1. 2020Supervised Contrastive Learningcited by 2,829
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  4. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,908
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  6. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  7. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  8. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  10. 2019Contrastive Multiview Codingcited by 1,301
  11. 2016Colorful Image Colorizationcited by 1,114
  12. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  13. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  14. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  15. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 657
  16. 2020Understanding the Behaviour of Contrastive Losscited by 398
  17. 2020Whitening for Self-Supervised Representation Learningcited by 279
  18. 2016Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)cited by 198
  19. 2021Decoupled Contrastive Learningcited by 143
  20. 2019Unsupervised Deep Learning by Neighbourhood Discoverycited by 78
  21. 2021A Petri Dish for Histopathology Image Analysiscited by 64
  22. 2021Hard Sample Aware Noise Robust Learning for Histopathology Image Classificationcited by 48
  23. 2023Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Datacited by 44
  24. 2016CliqueCNN: Deep Unsupervised Exemplar Learningcited by 41
  25. 2021Temperature as Uncertainty in Contrastive Learningcited by 25
  26. 2023Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualizationcited by 25
  27. 2017MURA: Large Dataset for Abnormality Detection in Musculoskeletal Radiographscited by 20
  28. 2022Model-Aware Contrastive Learning: Towards Escaping the Dilemmascited by 11
  29. 2023Interpreting and Correcting Medical Image Classification with PIP-Netcited by 11
  30. 2022From Modern CNNs to Vision Transformers: Assessing the Performance, Robustness, and Classification Strategies of Deep Learning Models in Histopathologycited by 9
  31. 2023Multi-Head Multi-Loss Model Calibrationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DROcited by 11
  2. 2025Subquadratic Algorithms and Hardness for Attention with Any Temperaturecited by 5
  3. 2026ViCLSR: A Supervised Contrastive Learning Framework with Natural Language Inference for Natural Language Understanding Taskscited by 0
  4. 2026TeMo: Temperature Modulation for Multimodal Contrastive Learningcited by 0