# Counterfactual Evaluation of Temporal Observation Protocols

arXiv: https://arxiv.org/abs/2608.22221 · HTML: https://mlsanity.com/paper/2608.22221/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 62 references that are in the map.
- First posted on arXiv 23 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions](https://mlsanity.com/paper/1502.06800/) (2015)
- [Modern Bayesian Experimental Design](https://mlsanity.com/paper/2302.14545/) (2023)
- [Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes](https://mlsanity.com/paper/1908.08526/) (2019)
- [External Validity: From Do-Calculus to Transportability Across Populations](https://mlsanity.com/paper/1503.01603/) (2015)
- [Off-policy Bandits with Deficient Support](https://mlsanity.com/paper/2006.09438/) (2020)
- [Graphical Models for Processing Missing Data](https://mlsanity.com/paper/1801.03583/) (2018)
- [A Survey on Principles, Models and Methods for Learning from Irregularly Sampled Time Series](https://mlsanity.com/paper/2012.00168/) (2020)
- [Probabilistic Integration: A Role in Statistical Computation?](https://mlsanity.com/paper/1512.00933/) (2015)
- [Full Law Identification In Graphical Models Of Missing Data: Completeness Results](https://mlsanity.com/paper/2004.04872/) (2020)
- [Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classification](https://mlsanity.com/paper/2202.00395/) (2022)
- [Goal-Oriented Optimal Design of Experiments for Large-Scale Bayesian Linear Inverse Problems](https://mlsanity.com/paper/1802.06517/) (2018)
- [Evaluation of Active Feature Acquisition Methods for Time-varying Feature Settings](https://mlsanity.com/paper/2312.01530/) (2023)
- [Optimal Designs for Longitudinal and Functional Data](https://mlsanity.com/paper/1604.05375/) (2016)
- [The Label Defines the Timescale: Trait-State Limits of Temporal-Aggregate Learning](https://mlsanity.com/paper/2608.01587/) (2026)

## Abstract (arXiv)

We study counterfactual protocol evaluation: whether data collected under a realised observation protocol determine the predictive value of alternatives that were never deployed. Protocol value is the population R2 of the Bayes-optimal predictor of a fixed trajectory-level target from the measurements an alternative would collect. We show that even infinite benchmark data need not determine this value: distinct latent covariance structures can induce the same benchmark measurement--target law while assigning different values to the same alternative. We develop a value-specific identification theory in which only latent ambiguity that changes the alternative's value matters. For linear targets, invisible covariance directions certify non-identification, while targeted measurements can restore identification without recovering the full latent covariance; an exact permutation construction extends the result to nonlinear aggregate targets. With finite dense calibration data, uniform error bounds control protocol-selection regret and distinguishable value gaps. Exact marginal gains then support cost-constrained, target-aware observation design. Simulations and retrospective analyses of Sleep-EDF and Long-Term AF show that broad temporal-layout differences can be more reliably distinguished than fine placements selected from finite data. Together, these results connect identification, calibration resolution and observation design for undeployed protocols.