mlsanityData to 7 Oct 2026

Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning

arXiv 2106.04015 ↗First posted 7 Jun 2021cs.LGcited by 81

Not on the map. Why?

Abstract (arXiv)

High-quality estimates of uncertainty and robustness are crucial for numerous real-world applications, especially for deep learning which underlies many deployed ML systems. The ability to compare techniques for improving these estimates is therefore very important for research and practice alike. Yet, competitive comparisons of methods are often lacking due to a range of reasons, including: compute availability for extensive tuning, incorporation of sufficiently many baselines, and concrete documentation for reproducibility. In this paper we introduce Uncertainty Baselines: high-quality implementations of standard and state-of-the-art deep learning methods on a variety of tasks. As of this writing, the collection spans 19 methods across 9 tasks, each with at least 5 metrics. Each baseline is a self-contained experiment pipeline with easily reusable and extendable components. Our goal is to provide immediate starting points for experimentation with new methods or applications. Additionally we provide model checkpoints, experiment outputs as Python notebooks, and leaderboards for comparing results. Code available at https://github.com/google/uncertainty-baselines.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 27 of its 49 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

27 of 49 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2016Wide Residual Networkscited by 3,057
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  11. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  12. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
15 more
  1. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  2. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  3. 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
  4. 2020Training independent subnetworks for robust predictioncited by 177
  5. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  6. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  7. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  8. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  9. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  10. 2020On Mixup Regularizationcited by 82
  11. 2019A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Taskscited by 62
  12. 2019Analyzing the Role of Model Uncertainty for Electronic Health Recordscited by 41
  13. 2019Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learningcited by 37
  14. 2021Correlated Input-Dependent Label Noise in Large-Scale Image Classificationcited by 37
  15. 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 27; 2023: 23; 2024: 12; 2025: 5; 2026: 4.

81 papers.

Sort:

By area:

  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  2. 2021Epistemic Neural Networkscited by 120
  3. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  4. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 100
  5. 2021Soft Calibration Objectives for Neural Networkscited by 89
  6. 2024Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Taskscited by 78
  7. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  8. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  9. 2021Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantificationcited by 57
  10. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  11. 2022Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Taskscited by 49
  12. 2022Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimationcited by 43
38 more
  1. 2023A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Modelscited by 39
  2. 2023What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifierscited by 35
  3. 2023On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Studycited by 25
  4. 2021The Neural Testbed: Evaluating Joint Predictionscited by 22
  5. 2022Shifts 2.0: Extending The Dataset of Real Distributional Shiftscited by 22
  6. 2022GFlowOut: Dropout with Generative Flow Networkscited by 22
  7. 2022Single Model Uncertainty Estimation via Stochastic Data Centeringcited by 20
  8. 2023Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shiftcited by 20
  9. 2023Function-Space Regularization in Neural Networks: A Probabilistic Perspectivecited by 20
  10. 2024Experts Don't Cheat: Learning What You Don't Know By Predicting Pairscited by 18
  11. 2022Is Self-Supervised Learning More Robust Than Supervised Learning?cited by 17
  12. 2022A Comprehensive Review of Digital Twin -- Part 2: Roles of Uncertainty Quantification and Optimization, a Battery Digital Twin, and Perspectivescited by 17
  13. 2023URL: A Representation Learning Benchmark for Transferable Uncertainty Estimatescited by 17
  14. 2023Fortuna: A Library for Uncertainty Quantification in Deep Learningcited by 14
  15. 2024End-to-End Conformal Calibration for Optimization Under Uncertaintycited by 14
  16. 2021Learning Augmentation Distributions using Transformed Risk Minimizationcited by 13
  17. 2022Prune and Tune Ensembles: Low-Cost Ensemble Learning With Sparse Independent Subnetworkscited by 13
  18. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
  19. 2022Ambiguous Images With Human Judgments for Robust Visual Event Classificationcited by 12
  20. 2023Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shiftscited by 12
  21. 2023Massively Scaling Heteroscedastic Classifierscited by 11
  22. 2023MUBen: Benchmarking the Uncertainty of Molecular Representation Modelscited by 11
  23. 2022Evaluating High-Order Predictive Distributions in Deep Learningcited by 10
  24. 2022Calibration and generalizability of probabilistic models on low-data chemical datasets with DIONYSUScited by 10
  25. 2023When does Privileged Information Explain Away Label Noise?cited by 10
  26. 2021Deep Classifiers with Label Noise Modeling and Distance Awarenesscited by 8
  27. 2022Task Bias in Vision-Language Modelscited by 8
  28. 2023On the role of Model Uncertainties in Bayesian Optimizationcited by 7
  29. 2024Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shiftscited by 7
  30. 2022Ensembling over Classifiers: a Bias-Variance Perspectivecited by 6
  31. 2022Sequential Bayesian Neural Subnetwork Ensemblescited by 5
  32. 2022Data Models for Dataset Drift Controls in Machine Learning With Optical Imagescited by 5
  33. 2023Implicit Variational Inference for High-Dimensional Posteriorscited by 5
  34. 2024Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forwardcited by 5
  35. 2024LUMA: A Benchmark Dataset for Learning from Uncertain and Multimodal Datacited by 5
  36. 2021A framework for benchmarking uncertainty in deep regressioncited by 4
  37. 2023Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networkscited by 4
  38. 2024Fast and reliable uncertainty quantification with neural network ensembles for industrial image classificationcited by 4

50 most cited of 81 All 81 (JSON)