# Few-shot Conformal Prediction with Auxiliary Tasks

arXiv: https://arxiv.org/abs/2102.08898 · HTML: https://mlsanity.com/paper/2102.08898/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 17 Feb 2021.

## Builds on (most cited first, 28 of 28)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://mlsanity.com/paper/2002.06470/) (2020)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [Towards a Neural Statistician](https://mlsanity.com/paper/1606.02185/) (2016)
- [Distributional conformal prediction](https://mlsanity.com/paper/1909.07889/) (2019)
- [Uncertainty Quantification Using Neural Networks for Molecular Property Prediction](https://mlsanity.com/paper/2005.10036/) (2020)
- [Few-shot Text Classification with Distributional Signatures](https://mlsanity.com/paper/1908.06039/) (2019)
- [Adaptive, Distribution-Free Prediction Intervals for Deep Networks](https://mlsanity.com/paper/1905.10634/) (2019)
- [With Malice Towards None: Assessing Uncertainty via Equalized Coverage](https://mlsanity.com/paper/1908.05428/) (2019)
- [Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction](https://mlsanity.com/paper/2004.10181/) (2020)

## Built on by (most cited first, 30 of 53)

- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [Conformal Language Modeling](https://mlsanity.com/paper/2306.10193/) (2023)
- [Consistent Accelerated Inference via Confident Adaptive Transformers](https://mlsanity.com/paper/2104.08803/) (2021)
- [Conformal Prediction for Deep Classifier via Label Ranking](https://mlsanity.com/paper/2310.06430/) (2023)
- [Conformal Prediction for Natural Language Processing: A Survey](https://mlsanity.com/paper/2405.01976/) (2024)
- [Federated Conformal Predictors for Distributed Uncertainty Quantification](https://mlsanity.com/paper/2305.17564/) (2023)
- [PAC Prediction Sets Under Covariate Shift](https://mlsanity.com/paper/2106.09848/) (2021)
- [Conformal Prediction Sets with Limited False Positives](https://mlsanity.com/paper/2202.07650/) (2022)
- [Conformal Prediction with Temporal Quantile Adjustments](https://mlsanity.com/paper/2205.09940/) (2022)
- [Conformal Prediction: A Data Perspective](https://mlsanity.com/paper/2410.06494/) (2024)
- [Split Localized Conformal Prediction](https://mlsanity.com/paper/2206.13092/) (2022)
- [Conformal prediction for text infilling and part-of-speech prediction](https://mlsanity.com/paper/2111.02592/) (2021)
- [Semantic uncertainty intervals for disentangled latent spaces](https://mlsanity.com/paper/2207.10074/) (2022)
- [Multivariate Conformal Prediction using Optimal Transport](https://mlsanity.com/paper/2502.03609/) (2025)
- [Non-Exchangeable Conformal Language Generation with Nearest Neighbors](https://mlsanity.com/paper/2402.00707/) (2024)
- [Stable Conformal Prediction Sets](https://mlsanity.com/paper/2112.10224/) (2021)
- [Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration](https://mlsanity.com/paper/2406.06818/) (2024)
- [Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction](https://mlsanity.com/paper/2210.03067/) (2022)
- [PAC Prediction Sets for Meta-Learning](https://mlsanity.com/paper/2207.02440/) (2022)
- [Conformal Autoregressive Generation: Beam Search with Coverage Guarantees](https://mlsanity.com/paper/2309.03797/) (2023)
- [Selective Generation for Controllable Language Models](https://mlsanity.com/paper/2307.09254/) (2023)
- [Recommendation Systems with Distribution-Free Reliability Guarantees](https://mlsanity.com/paper/2207.01609/) (2022)
- [On the Expected Size of Conformal Prediction Sets](https://mlsanity.com/paper/2306.07254/) (2023)
- [Semi-Supervised Risk Control via Prediction-Powered Inference](https://mlsanity.com/paper/2412.11174/) (2024)
- [Estimating Uncertainty in Multimodal Foundation Models using Public Internet Data](https://mlsanity.com/paper/2310.09926/) (2023)
- [Synthetic-Powered Predictive Inference](https://mlsanity.com/paper/2505.13432/) (2025)
- [Root-finding Approaches for Computing Conformal Prediction Set](https://mlsanity.com/paper/2104.06648/) (2021)
- [Knowing When to Stop: Delay-Adaptive Spiking Neural Network Classifiers with Reliability Guarantees](https://mlsanity.com/paper/2305.11322/) (2023)
- [Empirically Validating Conformal Prediction on Modern Vision Architectures Under Distribution Shift and Long-tailed Data](https://mlsanity.com/paper/2307.01088/) (2023)

## Abstract (arXiv)

We develop a novel approach to conformal prediction when the target task has limited data available for training. Conformal prediction identifies a small set of promising output candidates in place of a single prediction, with guarantees that the set contains the correct answer with high probability. When training data is limited, however, the predicted set can easily become unusably large. In this work, we obtain substantially tighter prediction sets while maintaining desirable marginal guarantees by casting conformal prediction as a meta-learning paradigm over exchangeable collections of auxiliary tasks. Our conformalization algorithm is simple, fast, and agnostic to the choice of underlying model, learning algorithm, or dataset. We demonstrate the effectiveness of this approach across a number of few-shot classification and regression tasks in natural language processing, computer vision, and computational chemistry for drug discovery.