mlsanityData to 14 Oct 2026

Model-Aware Contrastive Learning: Towards Escaping the Dilemmas

arXiv 2207.07874 ↗First posted 16 Jul 2022cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Contrastive learning (CL) continuously achieves significant breakthroughs across multiple domains. However, the most common InfoNCE-based methods suffer from some dilemmas, such as textituniformity-tolerance dilemma (UTD) and textitgradient reduction, both of which are related to a Pij term. It has been identified that UTD can lead to unexpected performance degradation. We argue that the fixity of temperature is to blame for UTD. To tackle this challenge, we enrich the CL loss family by presenting a Model-Aware Contrastive Learning (MACL) strategy, whose temperature is adaptive to the magnitude of alignment that reflects the basic confidence of the instance discrimination task, then enables CL loss to adjust the penalty strength for hard negatives adaptively. Regarding another dilemma, the gradient reduction issue, we derive the limits of an involved gradient scaling factor, which allows us to explain from a unified perspective why some recent approaches are effective with fewer negative samples, and summarily present a gradient reweighting to escape this dilemma. Extensive remarkable empirical results in vision, sentence, and graph modality validate our approach's general improvement for representation learning and downstream tasks.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 51 of its 63 references that are in the map. First posted on arXiv 16 Jul 2022.

Builds on

51 of 63 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2017Mask R-CNNcited by 7,277
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
38 more
  1. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  2. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  9. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  10. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  13. 2019Contrastive Multiview Codingcited by 1,301
  14. 2020Graph Contrastive Learning with Augmentationscited by 1,208
  15. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  16. 2019Strategies for Pre-training Graph Neural Networkscited by 971
  17. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  18. 2020TUDataset: A collection of benchmark datasets for learning with graphscited by 728
  19. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  20. 2020Contrastive Learning with Hard Negative Samplescited by 628
  21. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  22. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  23. 2020Understanding the Behaviour of Contrastive Losscited by 398
  24. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  25. 2020Debiased Contrastive Learningcited by 356
  26. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  27. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  28. 2021Graph Contrastive Learning Automatedcited by 287
  29. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 232
  30. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  31. 2021Decoupled Contrastive Learningcited by 143
  32. 2020Boosting Contrastive Self-Supervised Learning with False Negative Cancellationcited by 119
  33. 2019Unsupervised Deep Learning by Neighbourhood Discoverycited by 78
  34. 2022Let Invariant Rationale Discovery Inspire Graph Contrastive Learningcited by 57
  35. 2022Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCocited by 52
  36. 2019Are Powerful Graph Neural Nets Necessary? A Dissection on Graph Classificationcited by 39
  37. 2021Max-Margin Contrastive Learningcited by 27
  38. 2021Temperature as Uncertainty in Contrastive Learningcited by 25

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 2; 2026: 5.

11 papers.

  1. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 90
  2. 2024FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resourcescited by 9
  3. 2023Dynamically Scaled Temperature in Self-Supervised Contrastive Learningcited by 4
  4. 2026Taxonomy of the Retrieval System Framework: Pitfalls and Paradigmscited by 4
  5. 2025Molecular Graph Contrastive Learning with Line Graphcited by 3
  6. 2026Survival Reinforcement Learning: Toward Scalable Self-Supervised RLcited by 2
  7. 2024Improving Embedding Accuracy for Document Retrieval Using Entity Relationship Maps and Model-Aware Contrastive Samplingcited by 0
  8. 2025Understanding Self-supervised Contrastive Learning through Supervised Objectivescited by 0
  9. 2026Improving Domain Generalization in Contrastive Learning using Adaptive Temperature Controlcited by 0
  10. 2026Enhancing Language Models for Robust Greenwashing Detectioncited by 0
  11. 2026TeMo: Temperature Modulation for Multimodal Contrastive Learningcited by 0