# Conformal online model aggregation

arXiv: https://arxiv.org/abs/2403.15527 · HTML: https://mlsanity.com/paper/2403.15527/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 49 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 21 of 21)

- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [Adaptive Conformal Predictions for Time Series](https://mlsanity.com/paper/2202.07282/) (2022)
- [Conformal Inference for Online Prediction with Arbitrary Distribution Shifts](https://mlsanity.com/paper/2208.08401/) (2022)
- [Distribution-free uncertainty quantification for classification under label shift](https://mlsanity.com/paper/2103.03323/) (2021)
- [Conformal PID Control for Time Series Prediction](https://mlsanity.com/paper/2307.16895/) (2023)
- [Follow the Leader If You Can, Hedge If You Must](https://mlsanity.com/paper/1301.0534/) (2013)
- [Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data](https://mlsanity.com/paper/1802.06300/) (2018)
- [Online conformal prediction with decaying step sizes](https://mlsanity.com/paper/2402.01139/) (2024)
- [Selection and Aggregation of Conformal Prediction Sets](https://mlsanity.com/paper/2104.13871/) (2021)
- [Distribution-Free Prediction Sets for Two-Layer Hierarchical Models](https://mlsanity.com/paper/1809.07441/) (2018)
- [Merging uncertainty sets via majority vote](https://mlsanity.com/paper/2401.09379/) (2024)
- [One-Shot Federated Conformal Prediction](https://mlsanity.com/paper/2302.06322/) (2023)
- [Multi Split Conformal Prediction](https://mlsanity.com/paper/2103.00627/) (2021)
- [Randomized and Exchangeable Improvements of Markov's, Chebyshev's and Chernoff's Inequalities](https://mlsanity.com/paper/2304.02611/) (2023)
- [UTOPIA: Universally Trainable Optimal Prediction Intervals Aggregation](https://mlsanity.com/paper/2306.16549/) (2023)

## Built on by (most cited first, 11 of 11)

- [Conformal Prediction for Ensembles: Improving Efficiency via Score-Based Aggregation](https://mlsanity.com/paper/2405.16246/) (2024)
- [Parametric Scaling Law of Tuning Bias in Conformal Prediction](https://mlsanity.com/paper/2502.03023/) (2025)
- [Conformal Distributed Remote Inference in Sensor Networks Under Reliability and Communication Constraints](https://mlsanity.com/paper/2409.07902/) (2024)
- [Valid Selection among Conformal Sets](https://mlsanity.com/paper/2506.20173/) (2025)
- [Efficient Federated Conformal Prediction with Group-Conditional Guarantee](https://mlsanity.com/paper/2603.14198/) (2026)
- [JANET: Joint Adaptive predictioN-region Estimation for Time-series](https://mlsanity.com/paper/2407.06390/) (2024)
- [Multi-model Online Conformal Prediction with Graph-Structured Feedback](https://mlsanity.com/paper/2506.20898/) (2025)
- [CAOS: Conformal Aggregation of One-Shot Predictors](https://mlsanity.com/paper/2601.05219/) (2026)
- [Improving Coverage in Combined Prediction Sets with Weighted p-values](https://mlsanity.com/paper/2505.11785/) (2025)
- [Conformalized Super Learner](https://mlsanity.com/paper/2604.22391/) (2026)
- [Rolling Conformal Prediction in Sequential Model Training](https://mlsanity.com/paper/2609.26951/) (2026)

## Abstract (arXiv)

Conformal prediction equips machine learning models with a reasonable notion of uncertainty quantification without making strong distributional assumptions. It wraps around any prediction model and converts point predictions into set predictions with a predefined marginal coverage guarantee. However, conformal prediction only works if we fix the underlying machine learning model in advance. A relatively unaddressed issue in conformal prediction is that of model selection and/or aggregation: given a set of prediction models, which one should we conformalize? This paper suggests that instead of performing model selection, it can be prudent and practical to perform conformal set aggregation in an online, adaptive fashion. We propose a wrapper that takes in several conformal prediction sets (themselves wrapped around black-box prediction models), and outputs a single adaptively-combined prediction set. Our method, called conformal online model aggregation (COMA), is based on combining the prediction sets from several algorithms by weighted voting, and can be thought of as a sort of online stacking of the underlying conformal sets. As long as the input sets have (distribution-free) coverage guarantees, COMA retains coverage guarantees, under a negative correlation assumption between errors and weights. We verify that the assumption holds empirically in all settings considered. COMA is well-suited for decentralized or distributed settings, where different users may have different models, and are only willing to share their prediction sets for a new test point in a black-box fashion. As we demonstrate, it is also well-suited to settings with distribution drift and shift, where model selection can be imprudent.