# Distribution-Free Model-Agnostic Regression Calibration via Nonparametric Methods

arXiv: https://arxiv.org/abs/2305.12283 · HTML: https://mlsanity.com/paper/2305.12283/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 69 references that are in the map.
- First posted on arXiv 20 May 2023.

## Builds on (most cited first, 26 of 26)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Doubly Stochastic Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1705.08933/) (2017)
- [High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach](https://mlsanity.com/paper/1802.07167/) (2018)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification](https://mlsanity.com/paper/2011.09588/) (2020)
- [Non-Parametric Calibration for Classification](https://mlsanity.com/paper/1906.04933/) (2019)
- [Individual Calibration with Randomized Forecasting](https://mlsanity.com/paper/2006.10288/) (2020)
- [Improving Conditional Coverage via Orthogonal Quantile Regression](https://mlsanity.com/paper/2106.00394/) (2021)
- [Calibrated Reliable Regression using Maximum Mean Discrepancy](https://mlsanity.com/paper/2006.10255/) (2020)
- [Experiments with Random Projection](https://mlsanity.com/paper/1301.3849/) (2013)
- [Sequential estimation of quantiles with applications to A/B-testing and best-arm identification](https://mlsanity.com/paper/1906.09712/) (2019)
- [Best Arm Identification for Contaminated Bandits](https://mlsanity.com/paper/1802.09514/) (2018)
- [Local Calibration: Metrics and Recalibration](https://mlsanity.com/paper/2102.10809/) (2021)
- [Concentration bounds for empirical conditional value-at-risk: The unbounded case](https://mlsanity.com/paper/1808.01739/) (2018)
- [Building Calibrated Deep Models via Uncertainty Matching with Auxiliary Interval Predictors](https://mlsanity.com/paper/1909.04079/) (2019)
- [Conformalized Unconditional Quantile Regression](https://mlsanity.com/paper/2304.01426/) (2023)
- [X-Armed Bandits: Optimizing Quantiles, CVaR and Other Risks](https://mlsanity.com/paper/1904.08205/) (2019)
- [Quantile Bandits for Best Arms Identification](https://mlsanity.com/paper/2010.11568/) (2020)

## Built on by (most cited first, 3 of 3)

- [A Kernel-Based Conditional Two-Sample Test Using Nearest Neighbors (with Applications to Calibration, Regression Curves, and Simulation-Based Inference)](https://mlsanity.com/paper/2407.16550/) (2024)
- [When No-Rejection Learning is Consistent for Regression with Rejection](https://mlsanity.com/paper/2307.02932/) (2023)
- [Non-Asymptotic Analysis of Efficiency in Conformalized Regression](https://mlsanity.com/paper/2510.07093/) (2025)

## Abstract (arXiv)

In this paper, we consider the uncertainty quantification problem for regression models. Specifically, we consider an individual calibration objective for characterizing the quantiles of the prediction model. While such an objective is well-motivated from downstream tasks such as newsvendor cost, the existing methods have been largely heuristic and lack of statistical guarantee in terms of individual calibration. We show via simple examples that the existing methods focusing on population-level calibration guarantees such as average calibration or sharpness can lead to harmful and unexpected results. We propose simple nonparametric calibration methods that are agnostic of the underlying prediction model and enjoy both computational efficiency and statistical consistency. Our approach enables a better understanding of the possibility of individual calibration, and we establish matching upper and lower bounds for the calibration error of our proposed methods. Technically, our analysis combines the nonparametric analysis with a covering number argument for parametric analysis, which advances the existing theoretical analyses in the literature of nonparametric density estimation and quantile bandit problems. Importantly, the nonparametric perspective sheds new theoretical insights into regression calibration in terms of the curse of dimensionality and reconciles the existing results on the impossibility of individual calibration. To our knowledge, we make the first effort to reach both individual calibration and finite-sample guarantee with minimal assumptions in terms of conformal prediction. Numerical experiments show the advantage of such a simple approach under various metrics, and also under covariates shift. We hope our work provides a simple benchmark and a starting point of theoretical ground for future research on regression calibration.