mlsanityData to 14 Oct 2026

URL: A Representation Learning Benchmark for Transferable Uncertainty Estimates

arXiv 2307.03810 ↗First posted 7 Jul 2023cs.LG cs.AI stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

Representation learning has significantly driven the field to develop pretrained models that can act as a valuable starting point when transferring to new datasets. With the rising demand for reliable machine learning and uncertainty quantification, there is a need for pretrained models that not only provide embeddings but also transferable uncertainty estimates. To guide the development of such models, we propose the Uncertainty-aware Representation Learning (URL) benchmark. Besides the transferability of the representations, it also measures the zero-shot transferability of the uncertainty estimate using a novel metric. We apply URL to evaluate eleven uncertainty quantifiers that are pretrained on ImageNet and transferred to eight downstream datasets. We find that approaches that focus on the uncertainty of the representation itself or estimate the prediction risk directly outperform those that are based on the probabilities of upstream classes. Yet, achieving transferable uncertainty quantification remains an open challenge. Our findings indicate that it is not necessarily in conflict with traditional representation learning goals. Code is provided under https://github.com/mkirchhof/url .

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 45 of its 69 references that are in the map. First posted on arXiv 7 Jul 2023.

Builds on

45 of 69 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  9. 2020Supervised Contrastive Learningcited by 2,829
  10. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  11. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  12. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
33 more
  1. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  2. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  3. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  4. 2019Learning Loss for Active Learningcited by 336
  5. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 255
  6. 2019Human uncertainty makes classification more robustcited by 235
  7. 2018Hyperspherical Variational Auto-Encoderscited by 189
  8. 2019Probabilistic Face Embeddingscited by 173
  9. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  10. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 151
  11. 2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 139
  12. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 104
  13. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  14. 2020Revisiting Training Strategies and Generalization Performance in Deep Metric Learningcited by 100
  15. 2019Evaluating and Calibrating Uncertainty Prediction in Regression Taskscited by 87
  16. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 82
  17. 2023A Review of Uncertainty Estimation and its Application in Medical Imagingcited by 59
  18. 2020Combining Ensembles and Data Augmentation can Harm your Calibrationcited by 44
  19. 2022Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimationcited by 43
  20. 2021von Mises-Fisher Loss: An Exploration of Embedding Geometries for Supervised Learningcited by 42
  21. 2022ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCOcited by 37
  22. 2023What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifierscited by 35
  23. 2023A framework for benchmarking class-out-of-distribution detection and its application to ImageNetcited by 31
  24. 2023Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputscited by 29
  25. 2022A Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learningcited by 19
  26. 2023Learning Sample Difficulty from Pre-trained Models for Reliable Predictioncited by 15
  27. 2023Fortuna: A Library for Uncertainty Quantification in Deep Learningcited by 14
  28. 2023Massively Scaling Heteroscedastic Classifierscited by 11
  29. 2023When does Privileged Information Explain Away Label Noise?cited by 10
  30. 2022Probabilistic Embeddings Revisitedcited by 8
  31. 2022Uncertainty in Contrastive Learning: On the Predictability of Downstream Performancecited by 7
  32. 2023Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproductscited by 4
  33. 2022Towards Clear Expectations for Uncertainty Estimationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 5; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Taskscited by 78
  2. 2024Pretrained Visual Uncertaintiescited by 10
  3. 2025On the Importance of Embedding Norms in Self-Supervised Learningcited by 9
  4. 2024UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questionscited by 7
  5. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 4
  6. 2024Uncertainties of Latent Representations in Computer Visioncited by 1
  7. 2024Open-Set Semantic Uncertainty Aware Metric-Semantic Graph Matchingcited by 1
  8. 2024Legitimate ground-truth-free metrics for deep uncertainty classification scoringcited by 1
  9. 2025On the Generalization of Representation Uncertainty in Earth Observationcited by 1
  10. 2025Collision Risk Estimation via Loss Prediction in End-to-End Autonomous Drivingcited by 1
  11. 2026Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Modelscited by 1
  12. 2026Unstable Rankings in Bayesian Deep Learning Evaluationcited by 1
5 more
  1. 2026A Tale of Two Variances: When Single-Seed Benchmarks Fail in Bayesian Deep Learningcited by 1
  2. 2023Unveiling the Potential of Probabilistic Embeddings in Self-Supervised Learningcited by 0
  3. 2023A Central Motor System Inspired Pre-training Reinforcement Learning for Robotic Controlcited by 0
  4. 2024Improving OOD Generalization of Pre-trained Encoders via Aligned Embedding-Space Ensemblescited by 0
  5. 2025Influence of Classification Task and Distribution Shift Type on OOD Detection in Fetal Ultrasoundcited by 0