# When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making

arXiv: https://arxiv.org/abs/2011.06167 · HTML: https://mlsanity.com/paper/2011.06167/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2020.

## Built on by (most cited first, 12 of 12)

- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Predictive Multiplicity in Probabilistic Classification](https://mlsanity.com/paper/2206.01131/) (2022)
- [Uncertainty Estimation and Out-of-Distribution Detection for Counterfactual Explanations: Pitfalls and Solutions](https://mlsanity.com/paper/2107.09734/) (2021)
- [Assessing the communication gap between AI models and healthcare professionals: explainability, utility and trust in AI-driven clinical decision-making](https://mlsanity.com/paper/2204.05030/) (2022)
- [Algorithmic Assistance with Recommendation-Dependent Preferences](https://mlsanity.com/paper/2208.07626/) (2022)
- [Identifying Drivers of Predictive Aleatoric Uncertainty](https://mlsanity.com/paper/2312.07252/) (2023)
- [Uncertainty Quantification for Neurosymbolic Programs via Compositional Conformal Prediction](https://mlsanity.com/paper/2405.15912/) (2024)
- [Expanding Knowledge Graphs with Humans in the Loop](https://mlsanity.com/paper/2212.05189/) (2022)
- [Generating and Evaluating Explanations of Attended and Error-Inducing Input Regions for VQA Models](https://mlsanity.com/paper/2103.14712/) (2021)
- [Doubting AI Predictions: Influence-Driven Second Opinion Recommendation](https://mlsanity.com/paper/2205.00072/) (2022)
- [The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theories](https://mlsanity.com/paper/2501.12651/) (2025)
- [Underspecified Human Decision Experiments Considered Harmful](https://mlsanity.com/paper/2401.15106/) (2024)

## Abstract (arXiv)

As machine learning (ML) models are increasingly being employed to assist human decision makers, it becomes critical to provide these decision makers with relevant inputs which can help them decide if and how to incorporate model predictions into their decision making. For instance, communicating the uncertainty associated with model predictions could potentially be helpful in this regard. In this work, we carry out user studies (1,330 responses from 190 participants) to systematically assess how people with differing levels of expertise respond to different types of predictive uncertainty (i.e., posterior predictive distributions with different shapes and variances) in the context of ML assisted decision making for predicting apartment rental prices. We found that showing posterior predictive distributions led to smaller disagreements with the ML model's predictions, regardless of the shapes and variances of the posterior predictive distributions we considered, and that these effects may be sensitive to expertise in both ML and the domain. This suggests that posterior predictive distributions can potentially serve as useful decision aids which should be used with caution and take into account the type of distribution and the expertise of the human.