# On Second-Order Scoring Rules for Epistemic Uncertainty Quantification

arXiv: https://arxiv.org/abs/2301.12736 · HTML: https://mlsanity.com/paper/2301.12736/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 30 Jan 2023.

## Builds on (most cited first, 16 of 16)

- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts](https://mlsanity.com/paper/2006.09239/) (2020)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness](https://mlsanity.com/paper/1905.13472/) (2019)
- [Evidential Deep Learning for Open Set Action Recognition](https://mlsanity.com/paper/2107.10161/) (2021)
- [Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation](https://mlsanity.com/paper/2203.06102/) (2022)
- [The Unreasonable Effectiveness of Deep Evidential Regression](https://mlsanity.com/paper/2205.10060/) (2022)
- [Evaluating Robustness of Predictive Uncertainty Estimation: Are Dirichlet-based Models Reliable?](https://mlsanity.com/paper/2010.14986/) (2020)
- [Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions](https://mlsanity.com/paper/2111.08456/) (2021)
- [Regression Prior Networks](https://mlsanity.com/paper/2006.11590/) (2020)
- [Improving evidential deep learning via multi-task learning](https://mlsanity.com/paper/2112.09368/) (2021)
- [Evidential Conditional Neural Processes](https://mlsanity.com/paper/2212.00131/) (2022)

## Built on by (most cited first, 30 of 38)

- [A Survey on Uncertainty Quantification Methods for Deep Learning](https://mlsanity.com/paper/2302.13425/) (2023)
- [Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks](https://mlsanity.com/paper/2402.19460/) (2024)
- [Second-Order Uncertainty Quantification: A Distance-Based Approach](https://mlsanity.com/paper/2312.00995/) (2023)
- [Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?](https://mlsanity.com/paper/2402.09056/) (2024)
- [From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimation](https://mlsanity.com/paper/2402.10727/) (2024)
- [A Comprehensive Survey on Evidential Deep Learning and Its Applications](https://mlsanity.com/paper/2409.04720/) (2024)
- [Conformalized Credal Set Predictors](https://mlsanity.com/paper/2402.10723/) (2024)
- [Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs](https://mlsanity.com/paper/2402.08733/) (2024)
- [Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage?](https://mlsanity.com/paper/2402.06160/) (2024)
- [Revisiting Essential and Nonessential Settings of Evidential Deep Learning](https://mlsanity.com/paper/2410.00393/) (2024)
- [Uncertainty Estimation by Flexible Evidential Deep Learning](https://mlsanity.com/paper/2510.18322/) (2025)
- [Incentivizing honest performative predictions with proper scoring rules](https://mlsanity.com/paper/2305.17601/) (2023)
- [Conformal Prediction and Human Decision Making](https://mlsanity.com/paper/2503.11709/) (2025)
- [Dirichlet-based Uncertainty Quantification for Personalized Federated Learning with Improved Posterior Networks](https://mlsanity.com/paper/2312.11230/) (2023)
- [A calibration test for evaluating set-based epistemic uncertainty representations](https://mlsanity.com/paper/2502.16299/) (2025)
- [Learning Credal Ensembles via Distributionally Robust Optimization](https://mlsanity.com/paper/2602.08470/) (2026)
- [Reducing Aleatoric and Epistemic Uncertainty through Multi-modal Data Acquisition](https://mlsanity.com/paper/2501.18268/) (2025)
- [Probabilistic Self-supervised Learning via Scoring Rules Minimization](https://mlsanity.com/paper/2309.02048/) (2023)
- [Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review](https://mlsanity.com/paper/2504.18346/) (2025)
- [Linear Opinion Pooling for Uncertainty Quantification on Graphs](https://mlsanity.com/paper/2406.04041/) (2024)
- [Uncertainty-aware Evaluation of Auxiliary Anomalies with the Expected Anomaly Posterior](https://mlsanity.com/paper/2405.13699/) (2024)
- [Achieving Well-Informed Decision-Making in Drug Discovery: A Comprehensive Calibration Study using Neural Network-Based Structure-Activity Models](https://mlsanity.com/paper/2407.14185/) (2024)
- [Evidential Transformation Network: Turning Pretrained Models into Evidential Models for Post-hoc Uncertainty Estimation](https://mlsanity.com/paper/2604.08627/) (2026)
- [Evaluating Epistemic Uncertainty: Beyond OOD Detection and Active Learning](https://mlsanity.com/paper/2607.14817/) (2026)
- [Uncertainties of Latent Representations in Computer Vision](https://mlsanity.com/paper/2408.14281/) (2024)
- [CUQ-GNN: Committee-based Graph Uncertainty Quantification using Posterior Networks](https://mlsanity.com/paper/2409.04159/) (2024)
- [Joint Scoring Rules: Zero-Sum Competition Avoids Performative Prediction](https://mlsanity.com/paper/2412.20732/) (2024)
- [Efficient Credal Prediction through Decalibration](https://mlsanity.com/paper/2603.08495/) (2026)
- [Decision-Aligned Evaluation of Uncertainty Quantification](https://mlsanity.com/paper/2606.26990/) (2026)
- [Do you understand epistemic uncertainty? Think again! Rigorous frequentist epistemic uncertainty estimation in regression](https://mlsanity.com/paper/2503.13317/) (2025)

## Abstract (arXiv)

It is well known that accurate probabilistic predictors can be trained through empirical risk minimisation with proper scoring rules as loss functions. While such learners capture so-called aleatoric uncertainty of predictions, various machine learning methods have recently been developed with the goal to let the learner also represent its epistemic uncertainty, i.e., the uncertainty caused by a lack of knowledge and data. An emerging branch of the literature proposes the use of a second-order learner that provides predictions in terms of distributions on probability distributions. However, recent work has revealed serious theoretical shortcomings for second-order predictors based on loss minimisation. In this paper, we generalise these findings and prove a more fundamental result: There seems to be no loss function that provides an incentive for a second-order learner to faithfully represent its epistemic uncertainty in the same manner as proper scoring rules do for standard (first-order) learners. As a main mathematical tool to prove this result, we introduce the generalised notion of second-order scoring rules.