Cross-Domain Uncertainty Quantification for Selective Prediction: A Comprehensive Bound Ablation with Transfer-Informed Betting
Not on the map. Why?
Abstract (arXiv)
We present a comprehensive ablation of nine finite-sample bound families for selective prediction with risk control, combining concentration inequalities (Hoeffding, Empirical Bernstein, Clopper-Pearson, Wasserstein DRO, CVaR) with multiple-testing corrections (union bound, Learn Then Test fixed-sequence) and betting-based confidence sequences (WSR). Our main theoretical contribution is Transfer-Informed Betting (TIB), which warm-starts the WSR wealth process using a source domain's risk profile, achieving tighter bounds in data-scarce settings with a formal dominance guarantee. We prove that the TIB wealth process remains a valid supermartingale under all source-target divergences, that TIB dominates standard WSR when domains match, and that no data-independent warm-start can achieve better convergence. The combination of betting-based confidence sequences, LTT monotone testing, and cross-domain transfer is, to our knowledge, a three-way novelty not present in the literature. We evaluate all nine bound families on four benchmarks-MASSIVE (n=1,102), NyayaBench (n=280), CLINC-150 (n=22.5K), and Banking77 (n=13K)-across 18 (alpha, delta) configurations. On MASSIVE at alpha=0.10, LTT eliminates the ln(K) union-bound penalty, achieving 94.0% guaranteed coverage versus 73.8% for Hoeffding-a 27% relative improvement. On NyayaBench, where the small calibration set makes Hoeffding-family bounds infeasible below alpha=0.20, Transfer-Informed Betting achieves 18.5% coverage at alpha=0.10, a 5.4x improvement over LTT + Hoeffding. We additionally compare with split-conformal prediction, showing that conformal methods produce prediction sets (avg. 1.67 classes) whereas selective prediction provides single-prediction risk guarantees. We apply these methods to agentic caching systems, formalizing a progressive trust model where the guarantee determines when cached responses can be served autonomously.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 9 Mar 2026.
Builds on
16 of 23 references.
References · 16
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
- 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
- 2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
- 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
- 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 229
- 2020Estimating means of bounded random variables by bettingcited by 169
- 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
- 2022Efficient Few-Shot Learning Without Promptscited by 137
- 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
- 2021There are natural scores: Full comment on Shafer, "Testing by betting: A strategy for statistical and scientific communication"cited by 53
4 more
- 2025Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agentscited by 27
- 2025vCache: Verified Semantic Prompt Cachingcited by 17
- 2025Advancing Semantic Caching for LLMs with Domain-Specific Embeddings and Synthetic Datacited by 7
- 2026SemanticALLI: Caching Reasoning, Not Just Responses, in Agentic Systemscited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026FinAbstain: Uncertainty-Calibrated Multimodal RAG for Selective Financial Forecasting
- 2026Sign Betting for Misspecified Regression: Variance Adaptivity and Optimal Approximation
- 2025Second-Order Bounds for [0,1]-Valued Regression via Betting Loss
- 2023Risk Aware Benchmarking of Large Language Models
- 2026Available Guardrails: Certifying Selective Prediction across ML Systems
- 2026Taming the Greeks: Option Portfolios with Inductive Biases
- 2026Learning to Bet for Horizon-Aware Anytime-Valid Testing
- 2026FinBench: Time-Gated Calibration and Uncertainty Benchmarking for Agentic Financial Forecasting