mlsanityData to 14 Oct 2026

f-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning

arXiv 2402.10150 ↗First posted 15 Feb 2024cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

In self-supervised contrastive learning, a widely-adopted objective function is InfoNCE, which uses the heuristic cosine similarity for the representation comparison, and is closely related to maximizing the Kullback-Leibler (KL)-based mutual information. In this paper, we aim at answering two intriguing questions: (1) Can we go beyond the KL-based objective? (2) Besides the popular cosine similarity, can we design a better similarity function? We provide answers to both questions by generalizing the KL-based mutual information to the f-Mutual Information in Contrastive Learning (f-MICL) using the f-divergences. To answer the first question, we provide a wide range of f-MICL objectives which share the nice properties of InfoNCE (e.g., alignment and uniformity), and meanwhile result in similar or even superior performance. For the second question, assuming that the joint feature distribution is proportional to the Gaussian kernel, we derive an f-Gaussian similarity with better interpretability and empirical performance. Finally, we identify close relationships between the f-MICL objective and several popular InfoNCE-based objectives. Using benchmark tasks from both vision and natural language, we empirically evaluate f-MICL with different f-divergences on various architectures (SimCLR, MoCo, and MoCo v3) and datasets. We observe that f-MICL generally outperforms the benchmarks and the best-performing f-divergence is task and dataset dependent.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 39 of its 64 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

39 of 64 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
27 more
  1. 2016Density estimation using Real NVPcited by 2,042
  2. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  4. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  5. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  7. 2019Contrastive Multiview Codingcited by 1,301
  8. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  9. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  10. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  11. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  12. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  13. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  14. 2019On Variational Bounds of Mutual Informationcited by 559
  15. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  16. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  17. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  18. 2019On Mutual Information Maximization for Representation Learningcited by 316
  19. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  20. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 311
  21. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  22. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  23. 2020Boosting Contrastive Self-Supervised Learning with False Negative Cancellationcited by 119
  24. 2019Wasserstein Dependency Measure for Representation Learningcited by 69
  25. 2021f-Domain-Adversarial Learning: Theory and Algorithmscited by 47
  26. 2021Self-supervised Representation Learning with Relative Predictive Codingcited by 28
  27. 2022RenyiCL: Contrastive Representation Learning with Skew Renyi Divergencecited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024Indiscriminate Data Poisoning Attacks on Pre-trained Feature Extractorscited by 8
  2. 2025Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systemscited by 7
  3. 2024Characterizing Dependence of Samples along the Langevin Dynamics and Algorithms via Contraction of Phi-Mutual Informationcited by 5
  4. 2025A Statistical Theory of Contrastive Learning via Approximate Sufficient Statisticscited by 5
  5. 2025Global Minimizers of Sigmoid Contrastive Losscited by 5
  6. 2025Contrastive Predictive Coding Done Right for Mutual Information Estimationcited by 2
  7. 2024Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneitycited by 1
  8. 2024MUC: Machine Unlearning for Contrastive Learning with Black-box Evaluationcited by 1
  9. 2025Noisy-Pair Robust Representation Alignment for Positive-Unlabeled Learningcited by 1
  10. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  11. 2025Understanding InfoNCE: Transition Probability Matrix Induced Feature Clusteringcited by 0
  12. 2026Spectral Ghost in Representation Learning: from Component Analysis to Self-Supervised Learningcited by 0
1 more
  1. 2026When Softmax Fails at the Top: Extreme Value Corrections for InfoNCEcited by 0