# Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty

arXiv: https://arxiv.org/abs/2011.07586 · HTML: https://mlsanity.com/paper/2011.07586/

## Key facts

- Cited by 129 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2020.

## Built on by (most cited first, 30 of 129)

- [Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI](https://mlsanity.com/paper/2204.01075/) (2022)
- [Conformal Prediction with Large Language Models for Multi-Choice Question Answering](https://mlsanity.com/paper/2305.18404/) (2023)
- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling](https://mlsanity.com/paper/2311.08718/) (2023)
- [Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis](https://mlsanity.com/paper/2210.04714/) (2022)
- [Fair Conformal Predictors for Applications in Medical Imaging](https://mlsanity.com/paper/2109.04392/) (2021)
- [Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations](https://mlsanity.com/paper/2301.07255/) (2023)
- [Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation](https://mlsanity.com/paper/2110.03051/) (2021)
- [Evaluating Language Models for Mathematics through Interactions](https://mlsanity.com/paper/2306.01694/) (2023)
- ["I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust](https://mlsanity.com/paper/2405.00623/) (2024)
- [Calibrating Large Language Models Using Their Generations Only](https://mlsanity.com/paper/2403.05973/) (2024)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Uncertainty in Natural Language Generation: From Theory to Applications](https://mlsanity.com/paper/2307.15703/) (2023)
- [Socially Responsible AI Algorithms: Issues, Purposes, and Challenges](https://mlsanity.com/paper/2101.02032/) (2021)
- [Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning](https://mlsanity.com/paper/2202.05338/) (2022)
- [Investigating Explainability of Generative AI for Code through Scenario-based Design](https://mlsanity.com/paper/2202.04903/) (2022)
- [Federated Conformal Predictors for Distributed Uncertainty Quantification](https://mlsanity.com/paper/2305.17564/) (2023)
- [Eliciting and Learning with Soft Labels from Every Annotator](https://mlsanity.com/paper/2207.00810/) (2022)
- [Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Models](https://mlsanity.com/paper/2307.10236/) (2023)
- [Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI](https://mlsanity.com/paper/2206.10847/) (2022)
- [Human Uncertainty in Concept-Based AI Systems](https://mlsanity.com/paper/2303.12872/) (2023)
- [Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Sets](https://mlsanity.com/paper/2207.02238/) (2022)
- [Role of Human-AI Interaction in Selective Prediction](https://mlsanity.com/paper/2112.06751/) (2021)
- [Question-Driven Design Process for Explainable AI User Experiences](https://mlsanity.com/paper/2104.03483/) (2021)
- [Adapting the Linearised Laplace Model Evidence for Modern Deep Learning](https://mlsanity.com/paper/2206.08900/) (2022)
- [Human-Aligned Calibration for AI-Assisted Decision Making](https://mlsanity.com/paper/2306.00074/) (2023)
- [Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial](https://mlsanity.com/paper/2305.04933/) (2023)
- [Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity](https://mlsanity.com/paper/2210.15452/) (2022)
- [RELIC: Investigating Large Language Model Responses using Self-Consistency](https://mlsanity.com/paper/2311.16842/) (2023)
- [Uncertainty Quantification 360: A Holistic Toolkit for Quantifying and Communicating the Uncertainty of AI](https://mlsanity.com/paper/2106.01410/) (2021)

## Abstract (arXiv)

Algorithmic transparency entails exposing system properties to various stakeholders for purposes that include understanding, improving, and contesting predictions. Until now, most research into algorithmic transparency has predominantly focused on explainability. Explainability attempts to provide reasons for a machine learning model's behavior to stakeholders. However, understanding a model's specific behavior alone might not be enough for stakeholders to gauge whether the model is wrong or lacks sufficient knowledge to solve the task at hand. In this paper, we argue for considering a complementary form of transparency by estimating and communicating the uncertainty associated with model predictions. First, we discuss methods for assessing uncertainty. Then, we characterize how uncertainty can be used to mitigate model unfairness, augment decision-making, and build trustworthy systems. Finally, we outline methods for displaying uncertainty to stakeholders and recommend how to collect information required for incorporating uncertainty into existing ML pipelines. This work constitutes an interdisciplinary review drawn from literature spanning machine learning, visualization/HCI, design, decision-making, and fairness. We aim to encourage researchers and practitioners to measure, communicate, and use uncertainty as a form of transparency.