# Online Conformal Prediction Beyond Feedback

arXiv: https://arxiv.org/abs/2608.07139 · HTML: https://mlsanity.com/paper/2608.07139/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 22 of 22)

- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [MNIST-C: A Robustness Benchmark for Computer Vision](https://mlsanity.com/paper/1906.02337/) (2019)
- [LLMs Encode Harmfulness and Refusal Separately](https://mlsanity.com/paper/2507.11878/) (2025)
- [Improved Online Conformal Prediction via Strongly Adaptive Online Learning](https://mlsanity.com/paper/2302.07869/) (2023)
- [What Doubling Tricks Can and Can't Do for Multi-Armed Bandits](https://mlsanity.com/paper/1803.06971/) (2018)
- [Online conformal prediction with decaying step sizes](https://mlsanity.com/paper/2402.01139/) (2024)
- [Detecting High-Stakes Interactions with Activation Probes](https://mlsanity.com/paper/2506.10805/) (2025)
- [Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents](https://mlsanity.com/paper/2502.02561/) (2025)
- [Cleaning up the neighborhood: A full classification for adversarial partial monitoring](https://mlsanity.com/paper/1805.09247/) (2018)
- [The Relationship between No-Regret Learning and Online Conformal Prediction](https://mlsanity.com/paper/2502.10947/) (2025)
- [LLM Safety From Within: Detecting Harmful Content with Internal Representations](https://mlsanity.com/paper/2604.18519/) (2026)
- [Mirror Online Conformal Prediction with Intermittent Feedback](https://mlsanity.com/paper/2503.10345/) (2025)
- [Conformal Distributed Remote Inference in Sensor Networks Under Reliability and Communication Constraints](https://mlsanity.com/paper/2409.07902/) (2024)
- [Online Conformal Prediction with Adversarial Semi-bandit Feedback via Regret Minimization](https://mlsanity.com/paper/2604.17984/) (2026)
- [Efficient Online Set-valued Classification with Bandit Feedback](https://mlsanity.com/paper/2405.04393/) (2024)
- [Online Conformal Abstention Under Adversarial Bandit Feedback](https://mlsanity.com/paper/2506.14067/) (2025)
- [Calibrate-Then-Delegate: Safety Monitoring with Risk and Budget Guarantees via Model Cascades](https://mlsanity.com/paper/2604.14251/) (2026)
- [Staggered Integral Online Conformal Prediction for Safe Dynamics Adaptation with Multi-Step Coverage Guarantees](https://mlsanity.com/paper/2604.06058/) (2026)
- [Decision Making Needs Uncertainty Quantification [Lecture Notes]](https://mlsanity.com/paper/2607.14407/) (2026)

## Built on by (most cited first, 1 of 1)

- [Adaptive Conformal Inference Under Delayed Feedback: Coverage Guarantees and a Delay-to-Memory Diagnostic](https://mlsanity.com/paper/2609.07251/) (2026)

## Abstract (arXiv)

Online conformal prediction (OCP) provides theoretically principled uncertainty quantification for arbitrary black-box classifiers and non-i.i.d. data streams by constructing prediction sets that are guaranteed to contain the true label at a user-specified frequency. OCP usually updates prediction sets using feedback from previously deployed predictions. We instead study an OCP setting *beyond feedback*: on each round, the learner can *either* output a prediction set *or* query the correct label, but not both. Thus, no deployed prediction is ever evaluated directly. We reduce this problem to a partial monitoring game in which prediction actions return no observation and a separate query action reveals the label. To solve this game, we develop *OCP with queries* (OCPQ), a label-efficient forecasting algorithm designed for this beyond-feedback setting. Using an adequate reward construction, for any black-box classifier and any (non-i.i.d.) oblivious data stream of length T, OCPQ has O(T2/3) expected regret and expected coverage of at least β-O(T-1/3) for a user-defined β, while querying only an expected T-1/3 fraction of rounds. This provides coverage comparable to bandit-based OCP methods while requiring no feedback from deployed predictions. Experiments on real-world datasets demonstrate the effectiveness of our approach.