mlsanityData to 7 Oct 2026

Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

arXiv 1906.02530 ↗First posted 6 Jun 2019stat.ML cs.LGcited by 1,274

Not on the map. Why?

Cited by 27 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Modern machine learning methods including deep learning have achieved great success in predictive accuracy for supervised learning tasks, but may still fall short in giving useful estimates of their predictive em uncertainty. Quantifying uncertainty is especially critical in real-world settings, which often involve input distributions that are shifted from the training distribution due to a variety of factors including sample bias and non-stationarity. In such settings, well calibrated uncertainty estimates convey information about when a model's output should (or should not) be trusted. Many probabilistic deep learning methods, including Bayesian-and non-Bayesian methods, have been proposed in the literature for quantifying predictive uncertainty, but to our knowledge there has not previously been a rigorous large-scale empirical comparison of these methods under dataset shift. We present a large-scale benchmark of existing state-of-the-art methods on classification problems and investigate the effect of dataset shift on accuracy and calibration. We find that traditional post-hoc calibration does indeed fall short, as do several other previous methods. However, some methods that marginalize over models give surprisingly strong results across a broad spectrum of tasks.

Key facts. Cited by 1,274 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 209; 2022: 206; 2023: 227; 2024: 181; 2025: 176; 2026: 274.

1,274 papers.

Sort:

By area:

  1. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  3. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  4. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  7. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  9. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  10. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  12. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
38 more
  1. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 338
  2. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  3. 2021Unsolved Problems in ML Safetycited by 291
  4. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  5. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  6. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  7. 2021DICE: Leveraging Sparsification for Out-of-Distribution Detectioncited by 166
  8. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  9. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  10. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  11. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  12. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  13. 2023Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Surveycited by 140
  14. 2021PixMix: Dreamlike Pictures Comprehensively Improve Safety Measurescited by 131
  15. 2023The Science of Detecting LLM-Generated Textscited by 126
  16. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  17. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 124
  18. 2021Bayesian Neural Network Priors Revisitedcited by 121
  19. 2021Epistemic Neural Networkscited by 120
  20. 2021Predicting with Confidence on Unseen Distributionscited by 116
  21. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  22. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 113
  23. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  24. 2025Confidence Improves Self-Consistency in LLMscited by 111
  25. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  26. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  27. 2021AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptationcited by 104
  28. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  29. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  30. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 102
  31. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  32. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  33. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 100
  34. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  35. 2021The Devil is in the Margin: Margin-based Label Smoothing for Network Calibrationcited by 91
  36. 2022A high-resolution canopy height model of the Earthcited by 91
  37. 2021Soft Calibration Objectives for Neural Networkscited by 89
  38. 2021On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertaintycited by 87

50 most cited of 1,274 All 1,274 (JSON)