# Distribution-free risk assessment of regression-based machine learning algorithms

arXiv: https://arxiv.org/abs/2310.03545 · HTML: https://mlsanity.com/paper/2310.03545/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Least Ambiguous Set-Valued Classifiers with Bounded Error Levels](https://mlsanity.com/paper/1609.00451/) (2016)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [Distributional conformal prediction](https://mlsanity.com/paper/1909.07889/) (2019)
- [A Conformal Prediction Approach to Explore Functional Data](https://mlsanity.com/paper/1302.6452/) (2013)
- [A comparison of some conformal quantile regression methods](https://mlsanity.com/paper/1909.05433/) (2019)
- [Adaptive, Distribution-Free Prediction Intervals for Deep Networks](https://mlsanity.com/paper/1905.10634/) (2019)
- [Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Data](https://mlsanity.com/paper/2011.03274/) (2020)
- [Machine learning pipeline for battery state of health estimation](https://mlsanity.com/paper/2102.00837/) (2021)
- [JAWS: Auditing Predictive Uncertainty Under Covariate Shift](https://mlsanity.com/paper/2207.10716/) (2022)

## Built on by (most cited first, 6 of 6)

- [Conformal Policy Control](https://mlsanity.com/paper/2603.02196/) (2026)
- [Uncertainty quantification in metric spaces](https://mlsanity.com/paper/2405.05110/) (2024)
- [Calibrating Decision Robustness via Inverse Conformal Risk Control](https://mlsanity.com/paper/2510.07750/) (2025)
- [Conformalized Decision Risk Assessment](https://mlsanity.com/paper/2505.13243/) (2025)
- [Conformal and kNN Predictive Uncertainty Quantification Algorithms in Metric Spaces](https://mlsanity.com/paper/2507.15741/) (2025)
- [An In-Depth Examination of Risk Assessment in Multi-Class Classification Algorithms](https://mlsanity.com/paper/2412.04166/) (2024)

## Abstract (arXiv)

Machine learning algorithms have grown in sophistication over the years and are increasingly deployed for real-life applications. However, when using machine learning techniques in practical settings, particularly in high-risk applications such as medicine and engineering, obtaining the failure probability of the predictive model is critical. We refer to this problem as the risk-assessment task. We focus on regression algorithms and the risk-assessment task of computing the probability of the true label lying inside an interval defined around the model's prediction. We solve the risk-assessment problem using the conformal prediction approach, which provides prediction intervals that are guaranteed to contain the true label with a given probability. Using this coverage property, we prove that our approximated failure probability is conservative in the sense that it is not lower than the true failure probability of the ML algorithm. We conduct extensive experiments to empirically study the accuracy of the proposed method for problems with and without covariate shift. Our analysis focuses on different modeling regimes, dataset sizes, and conformal prediction methodologies.