# Representation Uncertainty in Self-Supervised Learning as Variational Inference

arXiv: https://arxiv.org/abs/2203.11437 · HTML: https://mlsanity.com/paper/2203.11437/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 62 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Hyperspherical Variational Auto-Encoders](https://mlsanity.com/paper/1804.00891/) (2018)
- [Contrastive Training for Improved Out-of-Distribution Detection](https://mlsanity.com/paper/2007.05566/) (2020)
- [On the uncertainty of self-supervised monocular depth estimation](https://mlsanity.com/paper/2005.06209/) (2020)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Bounding Box Regression with Uncertainty for Accurate Object Detection](https://mlsanity.com/paper/1809.08545/) (2018)
- [Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow](https://mlsanity.com/paper/1802.07095/) (2018)

## Built on by (most cited first, 22 of 22)

- [Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Driving](https://mlsanity.com/paper/2208.12263/) (2022)
- [UncTrack: Reliable Visual Object Tracking with Uncertainty-Aware Prototype Memory Network](https://mlsanity.com/paper/2503.12888/) (2025)
- [Pretrained Visual Uncertainties](https://mlsanity.com/paper/2402.16569/) (2024)
- [FreCT: Frequency-augmented Convolutional Transformer for Robust Time Series Anomaly Detection](https://mlsanity.com/paper/2505.00941/) (2025)
- [Collapse-Proof Non-Contrastive Self-Supervised Learning](https://mlsanity.com/paper/2410.04959/) (2024)
- [A Probabilistic Model Behind Self-Supervised Learning](https://mlsanity.com/paper/2402.01399/) (2024)
- [Understanding Self-Supervised Learning via Latent Distribution Matching](https://mlsanity.com/paper/2605.03517/) (2026)
- [Towards One Model for Classical Dimensionality Reduction: A Probabilistic Perspective on UMAP and t-SNE](https://mlsanity.com/paper/2405.17412/) (2024)
- [Multi-Token Enhancing for Vision Representation Learning](https://mlsanity.com/paper/2411.15787/) (2024)
- [DUAL: Dynamic Uncertainty-Aware Learning](https://mlsanity.com/paper/2506.03158/) (2025)
- [Transformers as Unrolled Inference in Probabilistic Laplacian Eigenmaps: An Interpretation and Potential Improvements](https://mlsanity.com/paper/2507.21040/) (2025)
- [Unifying Self-Supervised Clustering and Energy-Based Models](https://mlsanity.com/paper/2401.00873/) (2023)
- [Beyond Spectral Decomposition: Bayesian Contrastive Learning and its Non-negative Formulation via Factor Analysis](https://mlsanity.com/paper/2407.21740/) (2024)
- [SimSiam Naming Game: A Unified Approach for Emergent Communication and Representation Learning](https://mlsanity.com/paper/2410.21803/) (2024)
- [On the Generalization of Representation Uncertainty in Earth Observation](https://mlsanity.com/paper/2503.07082/) (2025)
- [Representation Synthesis by Probabilistic Many-Valued Logic Operation in Self-Supervised Learning](https://mlsanity.com/paper/2309.04148/) (2023)
- [Unveiling the Potential of Probabilistic Embeddings in Self-Supervised Learning](https://mlsanity.com/paper/2310.18080/) (2023)
- [Self-Supervised Learning with Gaussian Processes](https://mlsanity.com/paper/2512.09322/) (2025)
- [Variational Contrastive Learning for Skeleton-based Action Recognition](https://mlsanity.com/paper/2601.07666/) (2026)
- [Joint Embedding Variational Bayes](https://mlsanity.com/paper/2602.05639/) (2026)
- [Sensory Precision Inference for Multimodal Arbitration under Uncertainty](https://mlsanity.com/paper/2609.15065/) (2026)
- [Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisance](https://mlsanity.com/paper/2609.37789/) (2026)

## Abstract (arXiv)

In this study, a novel self-supervised learning (SSL) method is proposed, which considers SSL in terms of variational inference to learn not only representation but also representation uncertainties. SSL is a method of learning representations without labels by maximizing the similarity between image representations of different augmented views of an image. Meanwhile, variational autoencoder (VAE) is an unsupervised representation learning method that trains a probabilistic generative model with variational inference. Both VAE and SSL can learn representations without labels, but their relationship has not been investigated in the past. Herein, the theoretical relationship between SSL and variational inference has been clarified. Furthermore, a novel method, namely variational inference SimSiam (VI-SimSiam), has been proposed. VI-SimSiam can predict the representation uncertainty by interpreting SimSiam with variational inference and defining the latent space distribution. The present experiments qualitatively show that VI- SimSiam could learn uncertainty by comparing input images and predicted uncertainties. Additionally, we described a relationship between estimated uncertainty and classification accuracy.