# Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictions

arXiv: https://arxiv.org/abs/2212.13629 · HTML: https://mlsanity.com/paper/2212.13629/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 35 references that are in the map.
- First posted on arXiv 27 Dec 2022.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [TResNet: High Performance GPU-Dedicated Architecture](https://mlsanity.com/paper/2003.13630/) (2020)
- [Fairness risk measures](https://mlsanity.com/paper/1901.08665/) (2019)
- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Learning Bounds for Risk-sensitive Learning](https://mlsanity.com/paper/2006.08138/) (2020)
- [On the exact Berk-Jones statistics and their p-value calculation](https://mlsanity.com/paper/1311.3190/) (2013)
- [A New Confidence Interval for the Mean of a Bounded Random Variable](https://mlsanity.com/paper/1905.06208/) (2019)

## Built on by (most cited first, 14 of 14)

- [Fast yet Safe: Early-Exiting with Risk Control](https://mlsanity.com/paper/2405.20915/) (2024)
- [Prompt Risk Control: A Rigorous Framework for Responsible Deployment of Large Language Models](https://mlsanity.com/paper/2311.13628/) (2023)
- [Conformal Tail Risk Control for Large Language Model Alignment](https://mlsanity.com/paper/2502.20285/) (2025)
- [Distribution-Free Statistical Dispersion Control for Societal Applications](https://mlsanity.com/paper/2309.13786/) (2023)
- [Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization](https://mlsanity.com/paper/2407.17358/) (2024)
- [Conformal Prediction as Bayesian Quadrature](https://mlsanity.com/paper/2502.13228/) (2025)
- [Improving Predictor Reliability with Selective Recalibration](https://mlsanity.com/paper/2410.05407/) (2024)
- [QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictions](https://mlsanity.com/paper/2507.05220/) (2025)
- [Model-Free Assessment of Simulator Fidelity via Quantile Curves](https://mlsanity.com/paper/2512.05024/) (2025)
- [Adversarially Robust Control of Conditional Value-at-Risk via Rockafellar-Uryasev Conformal Inference](https://mlsanity.com/paper/2606.00320/) (2026)
- [Performative Risk Control: Calibrating Models for Reliable Deployment under Performativity](https://mlsanity.com/paper/2505.24097/) (2025)
- [Weighted Bayesian Conformal Prediction](https://mlsanity.com/paper/2604.06464/) (2026)
- [Occupancy-based Quantile Risk Control](https://mlsanity.com/paper/2609.03104/) (2026)
- [Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targets](https://mlsanity.com/paper/2609.27981/) (2026)

## Abstract (arXiv)

Rigorous guarantees about the performance of predictive algorithms are necessary in order to ensure their responsible use. Previous work has largely focused on bounding the expected loss of a predictor, but this is not sufficient in many risk-sensitive applications where the distribution of errors is important. In this work, we propose a flexible framework to produce a family of bounds on quantiles of the loss distribution incurred by a predictor. Our method takes advantage of the order statistics of the observed loss values rather than relying on the sample mean alone. We show that a quantile is an informative way of quantifying predictive performance, and that our framework applies to a variety of quantile-based metrics, each targeting important subsets of the data distribution. We analyze the theoretical properties of our proposed method and demonstrate its ability to rigorously control loss quantiles on several real-world datasets.