mlsanityData to 14 Oct 2026

The Neural Testbed: Evaluating Joint Predictions

arXiv 2110.04629 ↗First posted 9 Oct 2021cs.LG cs.AI stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

Predictive distributions quantify uncertainties ignored by point estimates. This paper introduces The Neural Testbed: an open-source benchmark for controlled and principled evaluation of agents that generate such predictions. Crucially, the testbed assesses agents not only on the quality of their marginal predictions per input, but also on their joint predictions across many inputs. We evaluate a range of agents using a simple neural network data generating process. Our results indicate that some popular Bayesian deep learning agents do not fare well with joint predictions, even when they can produce accurate marginal predictions. We also show that the quality of joint predictions drives performance in downstream decision tasks. We find these results are robust across choice a wide range of generative models, and highlight the practical importance of joint predictions to the community.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 21 of its 43 references that are in the map. First posted on arXiv 9 Oct 2021.

Builds on

21 of 43 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2015Variational Inference with Normalizing Flowscited by 2,170
  5. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  6. 2017A Tutorial on Thompson Samplingcited by 433
  7. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  8. 2021Deep learning: a statistical viewpointcited by 240
  9. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  10. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 200
  11. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 169
  12. 2019Functional Variational Bayesian Neural Networkscited by 127
9 more
  1. 2021Epistemic Neural Networkscited by 120
  2. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  3. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 82
  4. 2021Reinforcement Learning, Bit by Bitcited by 59
  5. 2015Bootstrapped Thompson Sampling and Deep Explorationcited by 40
  6. 2020Hypermodels for Explorationcited by 37
  7. 2021From Predictions to Decisions: The Importance of Joint Predictive Distributionscited by 29
  8. 2020Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?cited by 21
  9. 2017Variational Gaussian Dropout is not Bayesiancited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 4; 2025: 5; 2026: 2.

22 papers.

Sort:

By area:

  1. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  2. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 57
  3. 2024Efficient Exploration for LLMscited by 45
  4. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  5. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  6. 2022An Analysis of Ensemble Samplingcited by 24
  7. 2022Fine-Tuning Language Models via Epistemic Neural Networkscited by 22
  8. 2023Leveraging Demonstrations to Improve Online Learning: Quality Matterscited by 13
  9. 2023Scalable Neural Contextual Bandit for Recommender Systemscited by 12
  10. 2023Efficient Exploration via Epistemic-Risk-Seeking Policy Optimizationcited by 10
  11. 2024Making Better Use of Unlabelled Data in Bayesian Active Learningcited by 10
  12. 2023Reducing LLM Hallucinations using Epistemic Neural Networkscited by 8
10 more
  1. 2025Adaptive Elicitation of Latent Information Using Natural Languagecited by 7
  2. 2024The Need for a Big World Simulator: A Scientific Challenge for Continual Learningcited by 5
  3. 2025Architectural and Inferential Inductive Biases For Exchangeable Sequence Modelingcited by 2
  4. 2025SPIEDiff: robust learning of long-time macroscopic dynamics from short-time particle simulations with quantified epistemic uncertaintycited by 2
  5. 2023Transferring climate change physical knowledgecited by 1
  6. 2023PICProp: Physics-Informed Confidence Propagation for Uncertainty Quantificationcited by 1
  7. 2026Decision-Aligned Evaluation of Uncertainty Quantificationcited by 1
  8. 2025Prediction-Oriented Subsampling from Data Streamscited by 0
  9. 2025Active Learning with Task-Driven Representations for Messy Poolscited by 0
  10. 2026Subspace Inference Enables Efficient Active Reward Learning from Preferencescited by 0