# Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters

arXiv: https://arxiv.org/abs/2607.11653 · HTML: https://mlsanity.com/paper/2607.11653/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 43 references that are in the map.
- First posted on arXiv 13 Jul 2026.

## Builds on (most cited first, 17 of 17)

- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [Chronos-2: From Univariate to Universal Forecasting](https://mlsanity.com/paper/2510.15821/) (2025)
- [Estimating means of bounded random variables by betting](https://mlsanity.com/paper/2010.09686/) (2020)
- [Game-theoretic statistics and safe anytime-valid inference](https://mlsanity.com/paper/2210.01948/) (2022)
- [Coin Betting and Parameter-Free Online Learning](https://mlsanity.com/paper/1602.04128/) (2016)
- [Safe Testing](https://mlsanity.com/paper/1906.07801/) (2019)
- [River: machine learning for streaming data in Python](https://mlsanity.com/paper/2012.04740/) (2020)
- [Hypothesis testing with e-values](https://mlsanity.com/paper/2410.23614/) (2024)
- [Moirai 2.0: When Less Is More for Time Series Forecasting](https://mlsanity.com/paper/2511.11698/) (2025)
- [LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena](https://mlsanity.com/paper/2510.17638/) (2025)
- [Testing exchangeability: fork-convexity, supermartingales, and e-processes](https://mlsanity.com/paper/2102.00630/) (2021)
- [Sequential estimation of quantiles with applications to A/B-testing and best-arm identification](https://mlsanity.com/paper/1906.09712/) (2019)
- [E-backtesting](https://mlsanity.com/paper/2209.00991/) (2022)
- [Sequentially valid tests for forecast calibration](https://mlsanity.com/paper/2109.11761/) (2021)
- [Anytime-valid sequential testing for elicitable functionals via supermartingales](https://mlsanity.com/paper/2204.05680/) (2022)
- [Time-uniform confidence bands for the CDF under nonstationarity](https://mlsanity.com/paper/2302.14248/) (2023)

## Abstract (arXiv)

Black-box conditional quantile forecasts are widely used for sequential decisions under asymmetric costs, such as inventory planning in supply chain management. Once deployed, such forecasters must be monitored continuously as data streams drift and regimes change; this invalidates standard, fixed-horizon backtests for calibration. Further, existing backtests do not take into account that the notion of calibration is, in fact, information-dependent: forecasts can look calibrated to an auditor with coarse information while being miscalibrated to an auditor with richer information. We develop a distribution-free and game-theoretic testing framework for continuously auditing black-box conditional quantile forecasters with non-i.i.d. losses, such that the resulting evidence process is powerful against predictably chosen alternatives specified by the features available to the auditor. We first formalize notions of conditional quantile calibration when different sets of features are available to the auditor, establishing that the coarseness of the auditor's information set determines the hardness of the testing problem. We then identify the sets of alternatives for which the auditor can achieve power, and focusing on contextual bets linear in the features, we derive finite-time detection guarantees for such alternatives, all without an i.i.d. assumption. The resulting evidence processes are interpretable at the feature level, as they quantify fine-grained, "feature-aware" evidence for miscalibration. We empirically validate these methods on simulated and real data, finding that a popular time series forecaster (Chronos-2) is highly miscalibrated w.r.t. multiple relevant features.