# Uncertainty in Contrastive Learning: On the Predictability of Downstream Performance

arXiv: https://arxiv.org/abs/2207.09336 · HTML: https://mlsanity.com/paper/2207.09336/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 19 Jul 2022.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [A General Framework for Uncertainty Estimation in Deep Learning](https://mlsanity.com/paper/1907.06890/) (2019)
- [On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products](https://mlsanity.com/paper/1610.01256/) (2016)
- [Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks](https://mlsanity.com/paper/2102.12567/) (2021)
- [Temperature as Uncertainty in Contrastive Learning](https://mlsanity.com/paper/2110.04403/) (2021)
- [A Simple Framework for Uncertainty in Contrastive Learning](https://mlsanity.com/paper/2010.02038/) (2020)
- [Large-scale Distance Metric Learning with Uncertainty](https://mlsanity.com/paper/1805.10384/) (2018)

## Built on by (most cited first, 7 of 7)

- [Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputs](https://mlsanity.com/paper/2302.02865/) (2023)
- [URL: A Representation Learning Benchmark for Transferable Uncertainty Estimates](https://mlsanity.com/paper/2307.03810/) (2023)
- [Training-Free Uncertainty Estimation for Embedding Models](https://mlsanity.com/paper/2306.00206/) (2023)
- [Uni-SLAM: Uncertainty-Aware Neural Implicit SLAM for Real-Time Dense Indoor Scene Reconstruction](https://mlsanity.com/paper/2412.00242/) (2024)
- [Multi-Similarity Contrastive Learning](https://mlsanity.com/paper/2307.02712/) (2023)
- [Unveiling the Potential of Probabilistic Embeddings in Self-Supervised Learning](https://mlsanity.com/paper/2310.18080/) (2023)
- [FISHing in Uncertainty: Synthetic Contrastive Learning for Genetic Aberration Detection](https://mlsanity.com/paper/2411.01025/) (2024)

## Abstract (arXiv)

The superior performance of some of today's state-of-the-art deep learning models is to some extent owed to extensive (self-)supervised contrastive pretraining on large-scale datasets. In contrastive learning, the network is presented with pairs of positive (similar) and negative (dissimilar) datapoints and is trained to find an embedding vector for each datapoint, i.e., a representation, which can be further fine-tuned for various downstream tasks. In order to safely deploy these models in critical decision-making systems, it is crucial to equip them with a measure of their uncertainty or reliability. However, due to the pairwise nature of training a contrastive model, and the lack of absolute labels on the output (an abstract embedding vector), adapting conventional uncertainty estimation techniques to such models is non-trivial. In this work, we study whether the uncertainty of such a representation can be quantified for a single datapoint in a meaningful way. In other words, we explore if the downstream performance on a given datapoint is predictable, directly from its pre-trained embedding. We show that this goal can be achieved by directly estimating the distribution of the training data in the embedding space and accounting for the local consistency of the representations. Our experiments show that this notion of uncertainty for an embedding vector often strongly correlates with its downstream accuracy.