# CAOS: Conformal Aggregation of One-Shot Predictors

arXiv: https://arxiv.org/abs/2601.05219 · HTML: https://mlsanity.com/paper/2601.05219/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 36 references that are in the map.
- First posted on arXiv 8 Jan 2026.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [Conformal Language Modeling](https://mlsanity.com/paper/2306.10193/) (2023)
- [Theoretical Foundations of Conformal Prediction](https://mlsanity.com/paper/2411.11824/) (2024)
- [Selection by Prediction with Conformal p-values](https://mlsanity.com/paper/2210.01408/) (2022)
- [API Is Enough: Conformal Prediction for Large Language Models Without Logit-Access](https://mlsanity.com/paper/2403.01216/) (2024)
- [ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guarantees](https://mlsanity.com/paper/2407.00499/) (2024)
- [Few-shot Conformal Prediction with Auxiliary Tasks](https://mlsanity.com/paper/2102.08898/) (2021)
- [RAFT: A Real-World Few-Shot Text Classification Benchmark](https://mlsanity.com/paper/2109.14076/) (2021)
- [Merging uncertainty sets via majority vote](https://mlsanity.com/paper/2401.09379/) (2024)
- [Prune 'n Predict: Optimizing LLM Decision-making with Conformal Prediction](https://mlsanity.com/paper/2501.00555/) (2024)
- [Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction](https://mlsanity.com/paper/2210.03067/) (2022)
- [Conformal Prediction for Zero-Shot Models](https://mlsanity.com/paper/2505.24693/) (2025)
- [Optimized Conformal Selection: Powerful Selective Inference After Conformity Score Optimization](https://mlsanity.com/paper/2411.17983/) (2024)
- [Are foundation models for computer vision good conformal predictors?](https://mlsanity.com/paper/2412.06082/) (2024)
- [Conformal online model aggregation](https://mlsanity.com/paper/2403.15527/) (2024)
- [Conformal Prediction for Ensembles: Improving Efficiency via Score-Based Aggregation](https://mlsanity.com/paper/2405.16246/) (2024)
- [Synthetic-Powered Predictive Inference](https://mlsanity.com/paper/2505.13432/) (2025)
- [Trustworthy Few-Shot Transfer of Medical VLMs through Split Conformal Prediction](https://mlsanity.com/paper/2506.17503/) (2025)
- [TECP: Token-Entropy Conformal Prediction for LLMs](https://mlsanity.com/paper/2509.00461/) (2025)
- [Domain-Shift-Aware Conformal Prediction for Large Language Models](https://mlsanity.com/paper/2510.05566/) (2025)
- [Improving the statistical efficiency of cross-conformal prediction](https://mlsanity.com/paper/2503.01495/) (2025)
- [Valid Selection among Conformal Sets](https://mlsanity.com/paper/2506.20173/) (2025)
- [Symmetric Aggregation of Conformity Scores for Efficient Uncertainty Sets](https://mlsanity.com/paper/2512.06945/) (2025)
- [Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Prediction](https://mlsanity.com/paper/2504.17671/) (2025)

## Built on by (most cited first, 1 of 1)

- [Unifying Conformal Language Tasks with In-Context Ensembles](https://mlsanity.com/paper/2609.03005/) (2026)

## Abstract (arXiv)

One-shot prediction enables rapid adaptation of pretrained foundation models to new tasks using only one labeled example, but lacks principled uncertainty quantification. While conformal prediction provides finite-sample coverage guarantees, standard split conformal methods are inefficient in the one-shot setting due to data splitting and reliance on a single predictor. We propose Conformal Aggregation of One-Shot Predictors (CAOS), a conformal framework that adaptively aggregates multiple one-shot predictors and uses a leave-one-out calibration scheme to fully exploit scarce labeled data. Despite violating classical exchangeability assumptions, we prove that CAOS achieves valid marginal coverage using a monotonicity-based argument. Experiments on one-shot facial landmarking and RAFT text classification tasks show that CAOS produces substantially smaller prediction sets than split conformal baselines while maintaining reliable coverage.