# Evaluating High-Order Predictive Distributions in Deep Learning

arXiv: https://arxiv.org/abs/2202.13509 · HTML: https://mlsanity.com/paper/2202.13509/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 28 Feb 2022.

## Builds on (most cited first, 12 of 12)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [Bootstrapped Thompson Sampling and Deep Exploration](https://mlsanity.com/paper/1507.00300/) (2015)
- [Hypermodels for Exploration](https://mlsanity.com/paper/2006.07464/) (2020)
- [From Predictions to Decisions: The Importance of Joint Predictive Distributions](https://mlsanity.com/paper/2107.09224/) (2021)
- [Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?](https://mlsanity.com/paper/2011.03178/) (2020)

## Built on by (most cited first, 10 of 10)

- [Efficient Exploration for LLMs](https://mlsanity.com/paper/2402.00396/) (2024)
- [Fine-Tuning Language Models via Epistemic Neural Networks](https://mlsanity.com/paper/2211.01568/) (2022)
- [Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping](https://mlsanity.com/paper/2206.03633/) (2022)
- [Sampling-based inference for large linear models, with application to linearised Laplace](https://mlsanity.com/paper/2210.04994/) (2022)
- [Making Better Use of Unlabelled Data in Bayesian Active Learning](https://mlsanity.com/paper/2404.17249/) (2024)
- [Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitions](https://mlsanity.com/paper/2401.04305/) (2024)
- [Marginal and Joint Cross-Entropies & Predictives for Online Bayesian Inference, Active Learning, and Active Sampling](https://mlsanity.com/paper/2205.08766/) (2022)
- [Robustness of Epinets against Distributional Shifts](https://mlsanity.com/paper/2207.00137/) (2022)
- [Prediction-Oriented Subsampling from Data Streams](https://mlsanity.com/paper/2508.03868/) (2025)
- [Active Learning with Task-Driven Representations for Messy Pools](https://mlsanity.com/paper/2510.25926/) (2025)

## Abstract (arXiv)

Most work on supervised learning research has focused on marginal predictions. In decision problems, joint predictive distributions are essential for good performance. Previous work has developed methods for assessing low-order predictive distributions with inputs sampled i.i.d. from the testing distribution. With low-dimensional inputs, these methods distinguish agents that effectively estimate uncertainty from those that do not. We establish that the predictive distribution order required for such differentiation increases greatly with input dimension, rendering these methods impractical. To accommodate high-dimensional inputs, we introduce textitdyadic sampling, which focuses on predictive distributions associated with random textitpairs of inputs. We demonstrate that this approach efficiently distinguishes agents in high-dimensional examples involving simple logistic regression as well as complex synthetic and empirical data.