# Utility-Directed Conformal Prediction: A Decision-Aware Framework for Actionable Uncertainty Quantification

arXiv: https://arxiv.org/abs/2410.01767 · HTML: https://mlsanity.com/paper/2410.01767/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 23 of 23)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Smart "Predict, then Optimize"](https://mlsanity.com/paper/1710.08005/) (2017)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Least Ambiguous Set-Valued Classifiers with Bounded Error Levels](https://mlsanity.com/paper/1609.00451/) (2016)
- [Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization](https://mlsanity.com/paper/1809.05504/) (2018)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [Smart Predict-and-Optimize for Hard Combinatorial Optimization Problems](https://mlsanity.com/paper/1911.10092/) (2019)
- [A Survey of Contextual Optimization Methods for Decision Making under Uncertainty](https://mlsanity.com/paper/2306.10374/) (2023)
- [Fair Conformal Predictors for Applications in Medical Imaging](https://mlsanity.com/paper/2109.04392/) (2021)
- [Integrated Conditional Estimation-Optimization](https://mlsanity.com/paper/2110.12351/) (2021)
- [Conformal Contextual Robust Optimization](https://mlsanity.com/paper/2310.10003/) (2023)
- [Stochastic Optimization Forests](https://mlsanity.com/paper/2008.07473/) (2020)
- [End-to-end Conditional Robust Optimization](https://mlsanity.com/paper/2403.04670/) (2024)
- [End-to-End Conformal Calibration for Optimization Under Uncertainty](https://mlsanity.com/paper/2409.20534/) (2024)
- [Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise](https://mlsanity.com/paper/2406.03258/) (2024)
- [Decision-Focused Evaluation of Worst-Case Distribution Shift](https://mlsanity.com/paper/2407.03557/) (2024)
- [Utility-Based Abstraction and Categorization](https://mlsanity.com/paper/1303.1469/) (2013)

## Built on by (most cited first, 13 of 13)

- [Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents](https://mlsanity.com/paper/2502.02561/) (2025)
- [Conformal Risk Training: End-to-End Optimization of Conformal Risk Control](https://mlsanity.com/paper/2510.08748/) (2025)
- [Human-AI Collaborative Uncertainty Quantification](https://mlsanity.com/paper/2510.23476/) (2025)
- [Conformal Policy Control](https://mlsanity.com/paper/2603.02196/) (2026)
- [Calibrating Decision Robustness via Inverse Conformal Risk Control](https://mlsanity.com/paper/2510.07750/) (2025)
- [Multi-Round Human-AI Collaboration with User-Specified Requirements](https://mlsanity.com/paper/2602.17646/) (2026)
- [Conformal Risk-Averse Decision Making with Action Conditional Guarantee](https://mlsanity.com/paper/2606.05551/) (2026)
- [Learning Polyhedral Conformal Sets for Robust Optimization](https://mlsanity.com/paper/2605.08506/) (2026)
- [Distribution-Free Robust Predict-Then-Optimize in Function Spaces](https://mlsanity.com/paper/2602.08215/) (2026)
- [Decision-Calibrated Conformal Uncertainty for Pacing Decisions in Streaming Advertising](https://mlsanity.com/paper/2606.10187/) (2026)
- [Multivariate conformal uncertainty propagation in multitask atomistic simulation: Successes and pitfalls](https://mlsanity.com/paper/2609.31384/) (2026)
- [Threshold-Aware Conformal Routing](https://mlsanity.com/paper/2610.02487/) (2026)
- [Joint upper-bound coverage and route-choice utility: an empirical evaluation on two urban proxy tasks](https://mlsanity.com/paper/2610.06995/) (2026)

## Abstract (arXiv)

Interest has been growing in decision-focused machine learning methods which train models to account for how their predictions are used in downstream optimization problems. Doing so can often improve performance on subsequent decision problems. However, current methods for uncertainty quantification do not incorporate any information about downstream decisions. We develop a methodology based on conformal prediction to identify prediction sets that account for a downstream cost function, making them more appropriate to inform high-stakes decision-making. Our approach harnesses the strengths of conformal methods -- modularity, model-agnosticism, and statistical coverage guarantees -- while incorporating downstream decisions and user-specified utility functions. We prove that our methods retain standard coverage guarantees. Empirical evaluation across a range of datasets and utility metrics demonstrates that our methods achieve significantly lower costs than standard conformal methods. We present a real-world use case in healthcare diagnosis, where our method effectively incorporates the hierarchical structure of dermatological diseases. The method successfully generates sets with coherent diagnostic meaning, potentially aiding triage for dermatology diagnosis and illustrating how our method can ground high-stakes decision-making employing domain knowledge.