# Probabilistic Self-supervised Learning via Scoring Rules Minimization

arXiv: https://arxiv.org/abs/2309.02048 · HTML: https://mlsanity.com/paper/2309.02048/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 5 Sep 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Einsum Networks: Fast and Scalable Learning of Tractable Probabilistic Circuits](https://mlsanity.com/paper/2004.06231/) (2020)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [Better Uncertainty Calibration via Proper Scores for Classification and Beyond](https://mlsanity.com/paper/2203.07835/) (2022)
- [Instance-wise Hard Negative Example Generation for Contrastive Learning in Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2108.04547/) (2021)
- [On Second-Order Scoring Rules for Epistemic Uncertainty Quantification](https://mlsanity.com/paper/2301.12736/) (2023)
- [Probabilistic Forecasting with Generative Networks via Scoring Rule Minimization](https://mlsanity.com/paper/2112.08217/) (2021)
- [Deep Neural Networks as Point Estimates for Deep Gaussian Processes](https://mlsanity.com/paper/2105.04504/) (2021)
- [A 3D Probabilistic Deep Learning System for Detection and Diagnosis of Lung Cancer Using Low-Dose CT Scans](https://mlsanity.com/paper/1902.03233/) (2019)
- [Likelihood-Free Inference with Generative Neural Networks via Scoring Rule Minimization](https://mlsanity.com/paper/2205.15784/) (2022)
- [Self-Distilled Self-Supervised Representation Learning](https://mlsanity.com/paper/2111.12958/) (2021)
- [Online Learning with Continuous Ranked Probability Score](https://mlsanity.com/paper/1902.10173/) (2019)
- [Out of Distribution Detection on ImageNet-O](https://mlsanity.com/paper/2201.09352/) (2022)
- [Diversified Ensemble of Independent Sub-Networks for Robust Self-Supervised Representation Learning](https://mlsanity.com/paper/2308.14705/) (2023)

## Built on by (most cited first, 4 of 4)

- [Stochastic Vision Transformers with Wasserstein Distance-Aware Attention](https://mlsanity.com/paper/2311.18645/) (2023)
- [A Probabilistic Model for Non-Contrastive Learning](https://mlsanity.com/paper/2501.13031/) (2025)
- [CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignment](https://mlsanity.com/paper/2508.06434/) (2025)
- [Beyond Augmentation: Leveraging Inter-Instance Relation in Self-Supervised Representation Learning](https://mlsanity.com/paper/2510.22322/) (2025)

## Abstract (arXiv)

In this paper, we propose a novel probabilistic self-supervised learning via Scoring Rule Minimization (ProSMIN), which leverages the power of probabilistic models to enhance representation quality and mitigate collapsing representations. Our proposed approach involves two neural networks; the online network and the target network, which collaborate and learn the diverse distribution of representations from each other through knowledge distillation. By presenting the input samples in two augmented formats, the online network is trained to predict the target network representation of the same sample under a different augmented view. The two networks are trained via our new loss function based on proper scoring rules. We provide a theoretical justification for ProSMIN's convergence, demonstrating the strict propriety of its modified scoring rule. This insight validates the method's optimization process and contributes to its robustness and effectiveness in improving representation quality. We evaluate our probabilistic model on various downstream tasks, such as in-distribution generalization, out-of-distribution detection, dataset corruption, low-shot learning, and transfer learning. Our method achieves superior accuracy and calibration, surpassing the self-supervised baseline in a wide range of experiments on large-scale datasets like ImageNet-O and ImageNet-C, ProSMIN demonstrates its scalability and real-world applicability.