# Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification

arXiv: https://arxiv.org/abs/2011.09588 · HTML: https://mlsanity.com/paper/2011.09588/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2020.

## Built on by (most cited first, 30 of 78)

- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [DEUP: Direct Epistemic Uncertainty Prediction](https://mlsanity.com/paper/2102.08501/) (2021)
- [Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis](https://mlsanity.com/paper/2210.04714/) (2022)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [Better Uncertainty Calibration via Proper Scores for Classification and Beyond](https://mlsanity.com/paper/2203.07835/) (2022)
- [Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification](https://mlsanity.com/paper/2109.10254/) (2021)
- [Calibrated Multiple-Output Quantile Regression with Representation Learning](https://mlsanity.com/paper/2110.00816/) (2021)
- [Improving Conditional Coverage via Orthogonal Quantile Regression](https://mlsanity.com/paper/2106.00394/) (2021)
- [Achieving Risk Control in Online Learning Settings](https://mlsanity.com/paper/2205.09095/) (2022)
- [RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness](https://mlsanity.com/paper/2206.14502/) (2022)
- [A Large-Scale Study of Probabilistic Calibration in Neural Network Regression](https://mlsanity.com/paper/2306.02738/) (2023)
- [Calibrated Uncertainty Quantification for Operator Learning via Conformal Prediction](https://mlsanity.com/paper/2402.01960/) (2024)
- [BayesCap: Bayesian Identity Cap for Calibrated Uncertainty in Frozen Neural Networks](https://mlsanity.com/paper/2207.06873/) (2022)
- [Calibrating Multimodal Learning](https://mlsanity.com/paper/2306.01265/) (2023)
- [How to Evaluate Uncertainty Estimates in Machine Learning for Regression?](https://mlsanity.com/paper/2106.03395/) (2021)
- [Modular Conformal Calibration](https://mlsanity.com/paper/2206.11468/) (2022)
- [How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?](https://mlsanity.com/paper/2302.03679/) (2023)
- [Lightweight, Uncertainty-Aware Conformalized Visual Odometry](https://mlsanity.com/paper/2303.02207/) (2023)
- [Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity](https://mlsanity.com/paper/2309.06240/) (2023)
- [BAGEN: Are LLM Agents Budget-Aware?](https://mlsanity.com/paper/2606.00198/) (2026)
- [Generalizing Bayesian Optimization with Decision-theoretic Entropies](https://mlsanity.com/paper/2210.01383/) (2022)
- [Towards Instance-Wise Calibration: Local Amortized Diagnostics and Reshaping of Conditional Densities (LADaR)](https://mlsanity.com/paper/2205.14568/) (2022)
- [Understanding Pathologies of Deep Heteroskedastic Regression](https://mlsanity.com/paper/2306.16717/) (2023)
- [HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models](https://mlsanity.com/paper/2405.10299/) (2024)
- [Parametric and Multivariate Uncertainty Calibration for Regression and Object Detection](https://mlsanity.com/paper/2207.01242/) (2022)
- [Clarifying Trust of Materials Property Predictions using Neural Networks with Distribution-Specific Uncertainty Quantification](https://mlsanity.com/paper/2302.02595/) (2023)
- [Conformalized Unconditional Quantile Regression](https://mlsanity.com/paper/2304.01426/) (2023)
- [Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise](https://mlsanity.com/paper/2406.03258/) (2024)
- [Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shifts](https://mlsanity.com/paper/2403.05600/) (2024)
- [Uncertainty-aware Predict-Then-Optimize Framework for Equitable Post-Disaster Power Restoration](https://mlsanity.com/paper/2508.04780/) (2025)

## Abstract (arXiv)

Among the many ways of quantifying uncertainty in a regression setting, specifying the full quantile function is attractive, as quantiles are amenable to interpretation and evaluation. A model that predicts the true conditional quantiles for each input, at all quantile levels, presents a correct and efficient representation of the underlying uncertainty. To achieve this, many current quantile-based methods focus on optimizing the so-called pinball loss. However, this loss restricts the scope of applicable regression models, limits the ability to target many desirable properties (e.g. calibration, sharpness, centered intervals), and may produce poor conditional quantiles. In this work, we develop new quantile methods that address these shortcomings. In particular, we propose methods that can apply to any class of regression model, allow for selecting a trade-off between calibration and sharpness, optimize for calibration of centered intervals, and produce more accurate conditional quantiles. We provide a thorough experimental evaluation of our methods, which includes a high dimensional uncertainty quantification task in nuclear fusion.