# Improving Coverage in Combined Prediction Sets with Weighted p-values

arXiv: https://arxiv.org/abs/2505.11785 · HTML: https://mlsanity.com/paper/2505.11785/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 17 May 2025.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 23 of 23)

- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [Conformal Prediction With Conditional Guarantees](https://mlsanity.com/paper/2305.12616/) (2023)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [A comparison of some conformal quantile regression methods](https://mlsanity.com/paper/1909.05433/) (2019)
- [Selection and Aggregation of Conformal Prediction Sets](https://mlsanity.com/paper/2104.13871/) (2021)
- [Conformal prediction under ambiguous ground truth](https://mlsanity.com/paper/2307.09302/) (2023)
- [EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gate](https://mlsanity.com/paper/2112.14397/) (2021)
- [Merging uncertainty sets via majority vote](https://mlsanity.com/paper/2401.09379/) (2024)
- [Ensemble Conformalized Quantile Regression for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2202.08756/) (2022)
- [Conformal prediction with localization](https://mlsanity.com/paper/1908.08558/) (2019)
- [Multi Split Conformal Prediction](https://mlsanity.com/paper/2103.00627/) (2021)
- [Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structures](https://mlsanity.com/paper/1808.09011/) (2018)
- [Conformalized Fairness via Quantile Regression](https://mlsanity.com/paper/2210.02015/) (2022)
- [Integrating Uncertainty Awareness into Conformalized Quantile Regression](https://mlsanity.com/paper/2306.08693/) (2023)
- [Conformal prediction after data-dependent model selection](https://mlsanity.com/paper/2408.07066/) (2024)
- [Conformal online model aggregation](https://mlsanity.com/paper/2403.15527/) (2024)
- [Machine learning for industrial sensing and control: A survey and practical perspective](https://mlsanity.com/paper/2401.13836/) (2024)

## Abstract (arXiv)

Conformal prediction quantifies the uncertainty of machine learning models by augmenting point predictions with valid prediction sets. For complex scenarios involving multiple trials, models, or data sources, conformal prediction sets can be aggregated to create a prediction set that captures the overall uncertainty, often improving precision. However, aggregating multiple prediction sets with individual 1-α coverage inevitably weakens the overall guarantee, typically resulting in 1-2α worst-case coverage. In this work, we propose a framework for the weighted aggregation of prediction sets, where weights are assigned to each prediction set based on their contribution. Our framework offers flexible control over how the sets are aggregated, achieving tighter coverage bounds that interpolate between the 1-2α guarantee of the combined models and the 1-α guarantee of an individual model depending on the distribution of weights. Importantly, our framework generalizes to data-dependent weights, as we derive a procedure for weighted aggregation that maintains finite-sample validity even when the weights depend on the data. This extension makes our framework broadly applicable to settings where weights are learned, such as mixture-of-experts (MoE), and we demonstrate through experiments in the MoE setting that our methods achieve adaptive coverage.