mlsanityData to 14 Oct 2026

Understanding Contrastive Learning via Distributionally Robust Optimization

arXiv 2310.11048 ↗First posted 17 Oct 2023cs.LG cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

This study reveals the inherent tolerance of contrastive learning (CL) towards sampling bias, wherein negative samples may encompass similar semantics (eg labels). However, existing theories fall short in providing explanations for this phenomenon. We bridge this research gap by analyzing CL through the lens of distributionally robust optimization (DRO), yielding several key insights: (1) CL essentially conducts DRO over the negative sampling distribution, thus enabling robust performance across a variety of potential distributions and demonstrating robustness to sampling bias; (2) The design of the temperature τ is not merely heuristic but acts as a Lagrange Coefficient, regulating the size of the potential distribution set; (3) A theoretical connection is established between DRO and mutual information, thus presenting fresh evidence for ``InfoNCE as an estimate of MI'' and a new estimation approach for φ-divergence-based generalized mutual information. We also identify CL's potential shortcomings, including over-conservatism and sensitivity to outliers, and introduce a novel Adjusted InfoNCE loss (ADNCE) to mitigate these issues. It refines potential distribution, improving performance and accelerating convergence. Extensive experiments on various domains (image, sentence, and graphs) validate the effectiveness of the proposal. The code is available at urlhttps://github.com/junkangwu/ADNCE.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 46 of its 60 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

46 of 60 references.

References · 46

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  6. 2021Convex optimizationcited by 2,515
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  10. 2019Contrastive Multiview Codingcited by 1,301
  11. 2020Graph Contrastive Learning with Augmentationscited by 1,208
  12. 2020TUDataset: A collection of benchmark datasets for learning with graphscited by 728
34 more
  1. 2018MINE: Mutual Information Neural Estimationcited by 683
  2. 2020Contrastive Learning with Hard Negative Samplescited by 628
  3. 2019On Variational Bounds of Mutual Informationcited by 559
  4. 2020Understanding the Behaviour of Contrastive Losscited by 398
  5. 2020Debiased Contrastive Learningcited by 356
  6. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
  7. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  8. 2015Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulationscited by 291
  9. 2016Variance-based regularization with convex objectivescited by 155
  10. 2021Decoupled Contrastive Learningcited by 143
  11. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  12. 2015Distributionally Robust Logistic Regressioncited by 126
  13. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 123
  14. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  15. 2022Let Invariant Rationale Discovery Inspire Graph Contrastive Learningcited by 57
  16. 2021DORO: Distributional and Outlier Robust Optimizationcited by 56
  17. 2020Are all negatives created equal in contrastive instance discrimination?cited by 50
  18. 2021Incremental False Negative Detection for Contrastive Learningcited by 48
  19. 2022Investigating Why Contrastive Learning Benefits Robustness Against Label Noisecited by 46
  20. 2021Improving Contrastive Learning by Visualizing Feature Transformationcited by 44
  21. 2022Understanding Deep Contrastive Learning via Coordinate-wise Optimizationcited by 35
  22. 2020Learning Bounds for Risk-sensitive Learningcited by 33
  23. 2021Modeling the Second Player in Distributionally Robust Optimizationcited by 29
  24. 2021Self-supervised Representation Learning with Relative Predictive Codingcited by 28
  25. 2020Optimal Bounds between f-Divergences and Integral Probability Metricscited by 27
  26. 2019Distributionally Robust Counterfactual Risk Minimizationcited by 26
  27. 2021Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimizationcited by 18
  28. 2022Distributionally Robust Models with Parametric Likelihood Ratioscited by 17
  29. 2020Sampling-Decomposable Generative Adversarial Recommendercited by 16
  30. 2021Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Frameworkcited by 11
  31. 2022RenyiCL: Contrastive Representation Learning with Skew Renyi Divergencecited by 8
  32. 2017The discrete moment problem with nonconvex shape constraintscited by 6
  33. 2022Coresets for Wasserstein Distributionally Robust Optimization Problemscited by 6
  34. 2021Orthounimodal Distributionally Robust Optimization: Representation, Computation and Multivariate Extreme Event Applicationscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 8; 2026: 7.

21 papers.

Sort:

By area:

  1. 2025Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendationcited by 51
  2. 2024To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DROcited by 11
  3. 2024PSL: Rethinking and Improving Softmax Loss from Pairwise Perspective for Recommendationcited by 9
  4. 2023BSL: Understanding and Improving Softmax Loss for Recommendationcited by 5
  5. 2025An Augmentation Overlap Theory of Contrastive Learningcited by 5
  6. 2026Closing the Modality Gap Aligns Group-Wise Semanticscited by 5
  7. 2024Uncertainty Quantification on Graph Learning: A Surveycited by 4
  8. 2025Nested Stochastic Algorithm for Generalized Sinkhorn distance-Regularized Distributionally Robust Optimizationcited by 3
  9. 2025An Augmentation-Aware Theory for Self-Supervised Contrastive Learningcited by 3
  10. 2025Patch-Wise Hypergraph Contrastive Learning with Dual Normal Distribution Weighting for Multi-Domain Stain Transfercited by 2
  11. 2026OlmoEarth v1.2: A more efficient family of OlmoEarth modelscited by 2
  12. 2025Structure-preserving contrastive learning for spatial time seriescited by 1
9 more
  1. 2025FedSKC: Federated Learning with Non-IID Data via Structural Knowledge Collaborationcited by 1
  2. 2025Every Subtlety Counts: Fine-grained Person Independence Micro-Action Recognition via Distributionally Robust Optimizationcited by 1
  3. 2026Closing the gap in multimodal medical representation alignmentcited by 1
  4. 2024UDUC: An Uncertainty-driven Approach for Learning-based Robust Controlcited by 0
  5. 2024EMP: Enhance Memory in Data Pruningcited by 0
  6. 2026Correct and Weight: A Simple Yet Effective Loss for Implicit Feedback Recommendationcited by 0
  7. 2026Robust Biomedical Publication Type and Study Design Classification with Knowledge-Guided Perturbationscited by 0
  8. 2026Expressivity In Multimodal Contrastive Learningcited by 0
  9. 2026Beyond Empirical Support: Structured Outlier Generation via Sinkhorn Optimal Transportcited by 0