# f-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning

arXiv: https://arxiv.org/abs/2402.10150 · HTML: https://mlsanity.com/paper/2402.10150/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 64 references that are in the map.
- First posted on arXiv 15 Feb 2024.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)

## Built on by (most cited first, 13 of 13)

- [Indiscriminate Data Poisoning Attacks on Pre-trained Feature Extractors](https://mlsanity.com/paper/2402.12626/) (2024)
- [Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systems](https://mlsanity.com/paper/2505.18671/) (2025)
- [Characterizing Dependence of Samples along the Langevin Dynamics and Algorithms via Contraction of Phi-Mutual Information](https://mlsanity.com/paper/2402.17067/) (2024)
- [A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics](https://mlsanity.com/paper/2503.17538/) (2025)
- [Global Minimizers of Sigmoid Contrastive Loss](https://mlsanity.com/paper/2509.18552/) (2025)
- [Contrastive Predictive Coding Done Right for Mutual Information Estimation](https://mlsanity.com/paper/2510.25983/) (2025)
- [Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity](https://mlsanity.com/paper/2404.03854/) (2024)
- [MUC: Machine Unlearning for Contrastive Learning with Black-box Evaluation](https://mlsanity.com/paper/2406.03603/) (2024)
- [Noisy-Pair Robust Representation Alignment for Positive-Unlabeled Learning](https://mlsanity.com/paper/2510.01278/) (2025)
- [Debiased Model-based Representations for Sample-efficient Continuous Control](https://mlsanity.com/paper/2605.11711/) (2026)
- [Understanding InfoNCE: Transition Probability Matrix Induced Feature Clustering](https://mlsanity.com/paper/2511.12180/) (2025)
- [Spectral Ghost in Representation Learning: from Component Analysis to Self-Supervised Learning](https://mlsanity.com/paper/2601.20154/) (2026)
- [When Softmax Fails at the Top: Extreme Value Corrections for InfoNCE](https://mlsanity.com/paper/2606.00262/) (2026)

## Abstract (arXiv)

In self-supervised contrastive learning, a widely-adopted objective function is InfoNCE, which uses the heuristic cosine similarity for the representation comparison, and is closely related to maximizing the Kullback-Leibler (KL)-based mutual information. In this paper, we aim at answering two intriguing questions: (1) Can we go beyond the KL-based objective? (2) Besides the popular cosine similarity, can we design a better similarity function? We provide answers to both questions by generalizing the KL-based mutual information to the f-Mutual Information in Contrastive Learning (f-MICL) using the f-divergences. To answer the first question, we provide a wide range of f-MICL objectives which share the nice properties of InfoNCE (e.g., alignment and uniformity), and meanwhile result in similar or even superior performance. For the second question, assuming that the joint feature distribution is proportional to the Gaussian kernel, we derive an f-Gaussian similarity with better interpretability and empirical performance. Finally, we identify close relationships between the f-MICL objective and several popular InfoNCE-based objectives. Using benchmark tasks from both vision and natural language, we empirically evaluate f-MICL with different f-divergences on various architectures (SimCLR, MoCo, and MoCo v3) and datasets. We observe that f-MICL generally outperforms the benchmarks and the best-performing f-divergence is task and dataset dependent.