# Pretrained Visual Uncertainties

arXiv: https://arxiv.org/abs/2402.16569 · HTML: https://mlsanity.com/paper/2402.16569/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 54 references that are in the map.
- First posted on arXiv 26 Feb 2024.

## Builds on (most cited first, 30 of 32)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?](https://mlsanity.com/paper/2209.03302/) (2022)
- [DEUP: Direct Epistemic Uncertainty Prediction](https://mlsanity.com/paper/2102.08501/) (2021)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [Probabilistic Concept Bottleneck Models](https://mlsanity.com/paper/2306.01574/) (2023)
- [A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement](https://mlsanity.com/paper/2204.09308/) (2022)
- [On the Practicality of Deterministic Epistemic Uncertainty](https://mlsanity.com/paper/2107.00649/) (2021)
- [Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation](https://mlsanity.com/paper/2207.06214/) (2022)
- [Improved Probabilistic Image-Text Representations](https://mlsanity.com/paper/2305.18171/) (2023)
- [What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers](https://mlsanity.com/paper/2302.11874/) (2023)
- [A framework for benchmarking class-out-of-distribution detection and its application to ImageNet](https://mlsanity.com/paper/2302.11893/) (2023)
- [Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputs](https://mlsanity.com/paper/2302.02865/) (2023)
- [Representation Uncertainty in Self-Supervised Learning as Variational Inference](https://mlsanity.com/paper/2203.11437/) (2022)
- [A Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learning](https://mlsanity.com/paper/2207.03784/) (2022)
- [URL: A Representation Learning Benchmark for Transferable Uncertainty Estimates](https://mlsanity.com/paper/2307.03810/) (2023)
- [Learning Sample Difficulty from Pre-trained Models for Reliable Prediction](https://mlsanity.com/paper/2304.10127/) (2023)

## Built on by (most cited first, 10 of 10)

- [Scoring Rules and Calibration for Imprecise Probabilities](https://mlsanity.com/paper/2410.23001/) (2024)
- [ViLU: Learning Vision-Language Uncertainties for Failure Prediction](https://mlsanity.com/paper/2507.07620/) (2025)
- [Three Types of Calibration with Properties and their Semantic and Formal Relationships](https://mlsanity.com/paper/2504.18395/) (2025)
- [Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review](https://mlsanity.com/paper/2504.18346/) (2025)
- [Uncertainty Quantification for LLM Function-Calling](https://mlsanity.com/paper/2604.22985/) (2026)
- [Uncertainties of Latent Representations in Computer Vision](https://mlsanity.com/paper/2408.14281/) (2024)
- [Open-Set Semantic Uncertainty Aware Metric-Semantic Graph Matching](https://mlsanity.com/paper/2409.11555/) (2024)
- [On the Generalization of Representation Uncertainty in Earth Observation](https://mlsanity.com/paper/2503.07082/) (2025)
- [Collision Risk Estimation via Loss Prediction in End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.07425/) (2025)
- [Dynamic Distribution-Aware Uncertainty Tracking in Vision-Language Representation Learning](https://mlsanity.com/paper/2608.09011/) (2026)

## Abstract (arXiv)

Accurate uncertainty estimation is vital to trustworthy machine learning, yet uncertainties typically have to be learned for each task anew. This work introduces the first pretrained uncertainty modules for vision models. Similar to standard pretraining this enables the zero-shot transfer of uncertainties learned on a large pretraining dataset to specialized downstream datasets. We enable our large-scale pretraining on ImageNet-21k by solving a gradient conflict in previous uncertainty modules and accelerating the training by up to 180x. We find that the pretrained uncertainties generalize to unseen datasets. In scrutinizing the learned uncertainties, we find that they capture aleatoric uncertainty, disentangled from epistemic components. We demonstrate that this enables safe retrieval and uncertainty-aware dataset visualization. To encourage applications to further problems and domains, we release all pretrained checkpoints and code under https://github.com/mkirchhof/url .