# Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputs

arXiv: https://arxiv.org/abs/2302.02865 · HTML: https://mlsanity.com/paper/2302.02865/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 6 Feb 2023.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Hyperspherical Variational Auto-Encoders](https://mlsanity.com/paper/1804.00891/) (2018)
- [Probabilistic Face Embeddings](https://mlsanity.com/paper/1904.09658/) (2019)
- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [ProxyNCA++: Revisiting and Revitalizing Proxy Neighborhood Component Analysis](https://mlsanity.com/paper/2004.01113/) (2020)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagation](https://mlsanity.com/paper/2103.12605/) (2021)
- [Face Image Quality Assessment: A Literature Survey](https://mlsanity.com/paper/2009.01103/) (2020)
- [A Broad Study on the Transferability of Visual Representations with Contrastive Learning](https://mlsanity.com/paper/2103.13517/) (2021)
- [Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation](https://mlsanity.com/paper/2207.06214/) (2022)
- [von Mises-Fisher Loss: An Exploration of Embedding Geometries for Supervised Learning](https://mlsanity.com/paper/2103.15718/) (2021)
- [ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCO](https://mlsanity.com/paper/2204.03359/) (2022)
- [Disentanglement of Correlated Factors via Hausdorff Factorized Support](https://mlsanity.com/paper/2210.07347/) (2022)
- [Uncertainty Estimation in One-Stage Object Detection](https://mlsanity.com/paper/1905.10296/) (2019)
- [Revisiting the Transferability of Supervised Pretraining: an MLP Perspective](https://mlsanity.com/paper/2112.00496/) (2021)
- [Probabilistic Compositional Embeddings for Multimodal Image Retrieval](https://mlsanity.com/paper/2204.05845/) (2022)
- [Enriching ImageNet with Human Similarity Judgments and Psychological Embeddings](https://mlsanity.com/paper/2011.11015/) (2020)
- [A Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learning](https://mlsanity.com/paper/2207.03784/) (2022)
- [A Simple Framework for Uncertainty in Contrastive Learning](https://mlsanity.com/paper/2010.02038/) (2020)
- [Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise](https://mlsanity.com/paper/2301.01054/) (2023)
- [Probabilistic Embeddings Revisited](https://mlsanity.com/paper/2202.06768/) (2022)

## Built on by (most cited first, 29 of 29)

- [Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks](https://mlsanity.com/paper/2402.19460/) (2024)
- [Improved Probabilistic Image-Text Representations](https://mlsanity.com/paper/2305.18171/) (2023)
- [InfoNCE: Identifying the Gap Between Theory and Practice](https://mlsanity.com/paper/2407.00143/) (2024)
- [Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agents](https://mlsanity.com/paper/2505.22655/) (2025)
- [URL: A Representation Learning Benchmark for Transferable Uncertainty Estimates](https://mlsanity.com/paper/2307.03810/) (2023)
- [Identifiable Exchangeable Mechanisms for Causal Structure and Representation Learning](https://mlsanity.com/paper/2406.14302/) (2024)
- [Probabilistic Language-Image Pre-Training](https://mlsanity.com/paper/2410.18857/) (2024)
- [Pretrained Visual Uncertainties](https://mlsanity.com/paper/2402.16569/) (2024)
- [Fine-Grained Uncertainty Decomposition in Large Language Models: A Spectral Approach](https://mlsanity.com/paper/2509.22272/) (2025)
- [On the Importance of Embedding Norms in Self-Supervised Learning](https://mlsanity.com/paper/2502.09252/) (2025)
- [ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models](https://mlsanity.com/paper/2307.00398/) (2023)
- [Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review](https://mlsanity.com/paper/2504.18346/) (2025)
- [Understanding Self-Supervised Learning via Latent Distribution Matching](https://mlsanity.com/paper/2605.03517/) (2026)
- [Maximum Likelihood Learning of Latent Dynamics Without Reconstruction](https://mlsanity.com/paper/2505.23569/) (2025)
- [Beyond Spectral Decomposition: Bayesian Contrastive Learning and its Non-negative Formulation via Factor Analysis](https://mlsanity.com/paper/2407.21740/) (2024)
- [Probabilistic Variational Contrastive Learning](https://mlsanity.com/paper/2506.10159/) (2025)
- [ProbMed: A Probabilistic Framework for Medical Multimodal Binding](https://mlsanity.com/paper/2509.25711/) (2025)
- [Uncertainties of Latent Representations in Computer Vision](https://mlsanity.com/paper/2408.14281/) (2024)
- [On the Generalization of Representation Uncertainty in Earth Observation](https://mlsanity.com/paper/2503.07082/) (2025)
- [Performance of Conformal Prediction in Capturing Aleatoric Uncertainty](https://mlsanity.com/paper/2509.05826/) (2025)
- [UncertainGen: Uncertainty-Aware Representations of DNA Sequences for Metagenomic Binning](https://mlsanity.com/paper/2509.26116/) (2025)
- [Unveiling the Potential of Probabilistic Embeddings in Self-Supervised Learning](https://mlsanity.com/paper/2310.18080/) (2023)
- [Probabilistic Contrastive Learning with Explicit Concentration on the Hypersphere](https://mlsanity.com/paper/2405.16460/) (2024)
- [FISHing in Uncertainty: Synthetic Contrastive Learning for Genetic Aberration Detection](https://mlsanity.com/paper/2411.01025/) (2024)
- [Improving OOD Generalization of Pre-trained Encoders via Aligned Embedding-Space Ensembles](https://mlsanity.com/paper/2411.13073/) (2024)
- [The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning](https://mlsanity.com/paper/2606.04280/) (2026)
- [Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms](https://mlsanity.com/paper/2606.30625/) (2026)
- [Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models](https://mlsanity.com/paper/2608.22483/) (2026)
- [Contrastive Linearized Laplace: Loss Geometry, Valid Repairs, and Reliable Cores](https://mlsanity.com/paper/2610.14875/) (2026)

## Abstract (arXiv)

Contrastively trained encoders have recently been proven to invert the data-generating process: they encode each input, e.g., an image, into the true latent vector that generated the image (Zimmermann et al., 2021). However, real-world observations often have inherent ambiguities. For instance, images may be blurred or only show a 2D view of a 3D object, so multiple latents could have generated them. This makes the true posterior for the latent vector probabilistic with heteroscedastic uncertainty. In this setup, we extend the common InfoNCE objective and encoders to predict latent distributions instead of points. We prove that these distributions recover the correct posteriors of the data-generating process, including its level of aleatoric uncertainty, up to a rotation of the latent space. In addition to providing calibrated uncertainty estimates, these posteriors allow the computation of credible intervals in image retrieval. They comprise images with the same latent as a given query, subject to its uncertainty. Code is available at https://github.com/mkirchhof/Probabilistic_Contrastive_Learning