# Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning

arXiv: https://arxiv.org/abs/2410.09878 · HTML: https://mlsanity.com/paper/2410.09878/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 13 Oct 2024.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Least Ambiguous Set-Valued Classifiers with Bounded Error Levels](https://mlsanity.com/paper/1609.00451/) (2016)
- [Safe Planning in Dynamic Environments using Conformal Prediction](https://mlsanity.com/paper/2210.10254/) (2022)
- [Deep Partition Aggregation: Provable Defense against General Poisoning Attacks](https://mlsanity.com/paper/2006.14768/) (2020)
- [Certified Robustness to Label-Flipping Attacks via Randomized Smoothing](https://mlsanity.com/paper/2002.03018/) (2020)
- [Robust Validation: Confident Predictions Even When Distributions Shift](https://mlsanity.com/paper/2008.04267/) (2020)
- [Improved Certified Defenses against Data Poisoning with (Deterministic) Finite Aggregation](https://mlsanity.com/paper/2202.02628/) (2022)
- [Conformal Prediction for Time Series with Modern Hopfield Networks](https://mlsanity.com/paper/2303.12783/) (2023)
- [Merging uncertainty sets via majority vote](https://mlsanity.com/paper/2401.09379/) (2024)
- [Provably Robust Conformal Prediction with Improved Efficiency](https://mlsanity.com/paper/2404.19651/) (2024)
- [Probabilistically robust conformal prediction](https://mlsanity.com/paper/2307.16360/) (2023)
- [Multi Split Conformal Prediction](https://mlsanity.com/paper/2103.00627/) (2021)
- [Certified Robustness to Data Poisoning in Gradient-Based Training](https://mlsanity.com/paper/2406.05670/) (2024)
- [Run-Off Election: Improved Provable Defense against Data Poisoning Attacks](https://mlsanity.com/paper/2302.02300/) (2023)
- [Robust Yet Efficient Conformal Prediction Sets](https://mlsanity.com/paper/2407.09165/) (2024)
- [A Conformal Prediction Score that is Robust to Label Noise](https://mlsanity.com/paper/2405.02648/) (2024)
- [Verifiably Robust Conformal Prediction](https://mlsanity.com/paper/2405.18942/) (2024)
- [The Pitfalls and Promise of Conformal Inference Under Adversarial Attacks](https://mlsanity.com/paper/2405.08886/) (2024)
- [Certifiably Byzantine-Robust Federated Conformal Prediction](https://mlsanity.com/paper/2406.01960/) (2024)
- [ACon2: Adaptive Conformal Consensus for Provable Blockchain Oracles](https://mlsanity.com/paper/2211.09330/) (2022)

## Built on by (most cited first, 4 of 4)

- [Game-Theoretic Defenses for Robust Conformal Prediction Against Adversarial Attacks in Medical Imaging](https://mlsanity.com/paper/2411.04376/) (2024)
- [Ensuring Calibration Robustness in Split Conformal Prediction Under Adversarial Attacks](https://mlsanity.com/paper/2511.18562/) (2025)
- [Provable Robustness against Backdoor Attacks via the Primal-Dual Perspective on Differential Privacy](https://mlsanity.com/paper/2605.21780/) (2026)
- [Replicable Conformal Prediction](https://mlsanity.com/paper/2608.23638/) (2026)

## Abstract (arXiv)

Conformal prediction provides model-agnostic and distribution-free uncertainty quantification through prediction sets that are guaranteed to include the ground truth with any user-specified probability. Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data, which can significantly alter prediction sets in practice. As a solution, we propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning. To ensure reliability under training poisoning, we introduce smoothed score functions that reliably aggregate predictions of classifiers trained on distinct partitions of the training data. To ensure reliability under calibration poisoning, we construct multiple prediction sets, each calibrated on distinct subsets of the calibration data. We then aggregate them into a majority prediction set, which includes a class only if it appears in a majority of the individual sets. Both proposed aggregations mitigate the influence of datapoints in the training and calibration data on the final prediction set. We experimentally validate our approach on image classification tasks, achieving strong reliability while maintaining utility and preserving coverage on clean data. Overall, our approach represents an important step towards more trustworthy uncertainty quantification in the presence of data poisoning.