mlsanityData to 7 Oct 2026

On Second-Order Scoring Rules for Epistemic Uncertainty Quantification

arXiv 2301.12736 ↗First posted 30 Jan 2023cs.LG stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

It is well known that accurate probabilistic predictors can be trained through empirical risk minimisation with proper scoring rules as loss functions. While such learners capture so-called aleatoric uncertainty of predictions, various machine learning methods have recently been developed with the goal to let the learner also represent its epistemic uncertainty, i.e., the uncertainty caused by a lack of knowledge and data. An emerging branch of the literature proposes the use of a second-order learner that provides predictions in terms of distributions on probability distributions. However, recent work has revealed serious theoretical shortcomings for second-order predictors based on loss minimisation. In this paper, we generalise these findings and prove a more fundamental result: There seems to be no loss function that provides an incentive for a second-order learner to faithfully represent its epistemic uncertainty in the same manner as proper scoring rules do for standard (first-order) learners. As a main mathematical tool to prove this result, we introduce the generalised notion of second-order scoring rules.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 16 of its 29 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

16 of 29 references.

References · 16

  1. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  2. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  3. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 728
  4. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  5. 2019Deep Evidential Regressioncited by 354
  6. 2020Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Countscited by 163
  7. 2019Ensemble Distribution Distillationcited by 158
  8. 2019Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustnesscited by 104
  9. 2021Evidential Deep Learning for Open Set Action Recognitioncited by 96
  10. 2022Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisationcited by 65
  11. 2022The Unreasonable Effectiveness of Deep Evidential Regressioncited by 40
  12. 2020Evaluating Robustness of Predictive Uncertainty Estimation: Are Dirichlet-based Models Reliable?cited by 39
4 more
  1. 2021Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributionscited by 39
  2. 2020Regression Prior Networkscited by 28
  3. 2021Improving evidential deep learning via multi-task learningcited by 13
  4. 2022Evidential Conditional Neural Processescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 14; 2025: 9; 2026: 10.

38 papers.

Sort:

By area:

  1. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  2. 2024Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Taskscited by 78
  3. 2023Second-Order Uncertainty Quantification: A Distance-Based Approachcited by 35
  4. 2024Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?cited by 34
  5. 2024From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimationcited by 27
  6. 2024A Comprehensive Survey on Evidential Deep Learning and Its Applicationscited by 25
  7. 2024Conformalized Credal Set Predictorscited by 21
  8. 2024Experts Don't Cheat: Learning What You Don't Know By Predicting Pairscited by 18
  9. 2024Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage?cited by 17
  10. 2024Revisiting Essential and Nonessential Settings of Evidential Deep Learningcited by 15
  11. 2025Uncertainty Estimation by Flexible Evidential Deep Learningcited by 10
  12. 2023Incentivizing honest performative predictions with proper scoring rulescited by 9
26 more
  1. 2023Dirichlet-based Uncertainty Quantification for Personalized Federated Learning with Improved Posterior Networkscited by 8
  2. 2025Conformal Prediction and Human Decision Makingcited by 8
  3. 2025A calibration test for evaluating set-based epistemic uncertainty representationscited by 7
  4. 2026Learning Credal Ensembles via Distributionally Robust Optimizationcited by 6
  5. 2025Reducing Aleatoric and Epistemic Uncertainty through Multi-modal Data Acquisitioncited by 5
  6. 2023Probabilistic Self-supervised Learning via Scoring Rules Minimizationcited by 4
  7. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 3
  8. 2024Uncertainty-aware Evaluation of Auxiliary Anomalies with the Expected Anomaly Posteriorcited by 2
  9. 2024Linear Opinion Pooling for Uncertainty Quantification on Graphscited by 2
  10. 2024Achieving Well-Informed Decision-Making in Drug Discovery: A Comprehensive Calibration Study using Neural Network-Based Structure-Activity Modelscited by 2
  11. 2026Evidential Transformation Network: Turning Pretrained Models into Evidential Models for Post-hoc Uncertainty Estimationcited by 2
  12. 2024Uncertainties of Latent Representations in Computer Visioncited by 1
  13. 2024Joint Scoring Rules: Zero-Sum Competition Avoids Performative Predictioncited by 1
  14. 2026Decision-Aligned Evaluation of Uncertainty Quantificationcited by 1
  15. 2026Evaluating Epistemic Uncertainty: Beyond OOD Detection and Active Learningcited by 1
  16. 2024CUQ-GNN: Committee-based Graph Uncertainty Quantification using Posterior Networkscited by 0
  17. 2025Do you understand epistemic uncertainty? Think again! Rigorous frequentist epistemic uncertainty estimation in regressioncited by 0
  18. 2025Detecting underdetermination in parameterized quantum circuitscited by 0
  19. 2025Post-Hoc Split-Point Self-Consistency Verification for Efficient, Unified Quantification of Aleatoric and Epistemic Uncertainty in Deep Learningcited by 0
  20. 2025Uncertainty-Aware Calibrated Clinical Text Classification with Large Language Modelscited by 0
  21. 2026Density-Informed Pseudo-Counts for Calibrated Evidential Deep Learningcited by 0
  22. 2026Efficient Credal Prediction through Decalibrationcited by 0
  23. 2026Courtroom Analogy: New Perspective on Uncertainty-Aware Classificationcited by 0
  24. 2026Can we trust our models? Epistemic calibration in second-order classificationcited by 0
  25. 2026It depends: Incorporating correlations for joint aleatoric and epistemic uncertainties of high-dimensional output spacescited by 0
  26. 2026Flexible Spectral-Normalized Neural Gaussian Process for Dynamic Aperture Predictioncited by 0