# Learning Optimal Conformal Classifiers

arXiv: https://arxiv.org/abs/2110.09192 · HTML: https://mlsanity.com/paper/2110.09192/

## Key facts

- Cited by 112 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 18 Oct 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [Least Ambiguous Set-Valued Classifiers with Bounded Error Levels](https://mlsanity.com/paper/1609.00451/) (2016)
- [EMNIST: an extension of MNIST to handwritten letters](https://mlsanity.com/paper/1702.05373/) (2017)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [Fast Differentiable Sorting and Ranking](https://mlsanity.com/paper/2002.08871/) (2020)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [A deep learning system for differential diagnosis of skin diseases](https://mlsanity.com/paper/1909.05382/) (2019)
- [Differentiable Ranks and Sorting using Optimal Transport](https://mlsanity.com/paper/1905.11885/) (2019)
- [A Conformal Prediction Approach to Explore Functional Data](https://mlsanity.com/paper/1302.6452/) (2013)
- [Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions](https://mlsanity.com/paper/2104.03829/) (2021)
- [Cautious Deep Learning](https://mlsanity.com/paper/1805.09460/) (2018)
- [Optimized conformal classification using gradient descent approximation](https://mlsanity.com/paper/2105.11255/) (2021)

## Built on by (most cited first, 30 of 112)

- [Large language model validity via enhanced conformal prediction methods](https://mlsanity.com/paper/2406.09714/) (2024)
- [Uncertainty Quantification over Graph with Conformalized Graph Neural Networks](https://mlsanity.com/paper/2305.14535/) (2023)
- [Improved Online Conformal Prediction via Strongly Adaptive Online Learning](https://mlsanity.com/paper/2302.07869/) (2023)
- [Training Uncertainty-Aware Classifiers with Conformalized Deep Learning](https://mlsanity.com/paper/2205.05878/) (2022)
- [Conformal Prediction for Deep Classifier via Label Ranking](https://mlsanity.com/paper/2310.06430/) (2023)
- [Length Optimization in Conformal Prediction](https://mlsanity.com/paper/2406.18814/) (2024)
- [On the Utility of Prediction Sets in Human-AI Teams](https://mlsanity.com/paper/2205.01411/) (2022)
- [Predictive Inference with Feature Conformal Prediction](https://mlsanity.com/paper/2210.00173/) (2022)
- [Object Pose Estimation with Statistical Guarantees: Conformal Keypoint Detection and Geometric Uncertainty Propagation](https://mlsanity.com/paper/2303.12246/) (2023)
- [Conformal Prediction Regions for Time Series using Linear Complementarity Programming](https://mlsanity.com/paper/2304.01075/) (2023)
- [Conformal prediction under ambiguous ground truth](https://mlsanity.com/paper/2307.09302/) (2023)
- [Bayesian Optimization with Conformal Prediction Sets](https://mlsanity.com/paper/2210.12496/) (2022)
- [Distribution Free Prediction Sets for Node Classification](https://mlsanity.com/paper/2211.14555/) (2022)
- [Provably Robust Conformal Prediction with Improved Efficiency](https://mlsanity.com/paper/2404.19651/) (2024)
- [Boosted Conformal Prediction Intervals](https://mlsanity.com/paper/2406.07449/) (2024)
- [E-Values Expand the Scope of Conformal Prediction](https://mlsanity.com/paper/2503.13050/) (2025)
- [Conformal Off-Policy Prediction in Contextual Bandits](https://mlsanity.com/paper/2206.04405/) (2022)
- [Split Localized Conformal Prediction](https://mlsanity.com/paper/2206.13092/) (2022)
- [Conformalized Credal Set Predictors](https://mlsanity.com/paper/2402.10723/) (2024)
- [An Information Theoretic Perspective on Conformal Prediction](https://mlsanity.com/paper/2405.02140/) (2024)
- [Forking Uncertainties: Reliable Prediction and Model Predictive Control with Sequence Models via Conformal Risk Control](https://mlsanity.com/paper/2310.10299/) (2023)
- [Trustworthy Classification through Rank-Based Conformal Prediction Sets](https://mlsanity.com/paper/2407.04407/) (2024)
- [Prune 'n Predict: Optimizing LLM Decision-making with Conformal Prediction](https://mlsanity.com/paper/2501.00555/) (2024)
- [Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction](https://mlsanity.com/paper/2210.03067/) (2022)
- [Robust Yet Efficient Conformal Prediction Sets](https://mlsanity.com/paper/2407.09165/) (2024)
- [End-to-End Conformal Calibration for Optimization Under Uncertainty](https://mlsanity.com/paper/2409.20534/) (2024)
- [Wasserstein-regularized Conformal Prediction under General Distribution Shift](https://mlsanity.com/paper/2501.13430/) (2025)
- [Conformal Prediction for Zero-Shot Models](https://mlsanity.com/paper/2505.24693/) (2025)
- [Conformal inference is (almost) free for neural networks trained with early stopping](https://mlsanity.com/paper/2301.11556/) (2023)
- [Optimized Conformal Selection: Powerful Selective Inference After Conformity Score Optimization](https://mlsanity.com/paper/2411.17983/) (2024)

## Abstract (arXiv)

Modern deep learning based classifiers show very high accuracy on test data but this does not provide sufficient guarantees for safe deployment, especially in high-stake AI applications such as medical diagnosis. Usually, predictions are obtained without a reliable uncertainty estimate or a formal guarantee. Conformal prediction (CP) addresses these issues by using the classifier's predictions, e.g., its probability estimates, to predict confidence sets containing the true class with a user-specified probability. However, using CP as a separate processing step after training prevents the underlying model from adapting to the prediction of confidence sets. Thus, this paper explores strategies to differentiate through CP during training with the goal of training model with the conformal wrapper end-to-end. In our approach, conformal training (ConfTr), we specifically "simulate" conformalization on mini-batches during training. Compared to standard training, ConfTr reduces the average confidence set size (inefficiency) of state-of-the-art CP methods applied after training. Moreover, it allows to "shape" the confidence sets predicted at test time, which is difficult for standard CP. On experiments with several datasets, we show ConfTr can influence how inefficiency is distributed across classes, or guide the composition of confidence sets in terms of the included classes, while retaining the guarantees offered by CP.