# Are We Measuring Anticipation? Auditing Privileged Information in Procedural Video Evaluation

arXiv: https://arxiv.org/abs/2610.03826 · HTML: https://mlsanity.com/paper/2610.03826/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 2 Oct 2026.

## Builds on (most cited first, 12 of 12)

- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [When will you do what? - Anticipating Temporal Occurrences of Activities](https://mlsanity.com/paper/1804.00892/) (2018)
- [What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention](https://mlsanity.com/paper/1905.09035/) (2019)
- [AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?](https://mlsanity.com/paper/2307.16368/) (2023)
- [Future Transformer for Long-term Action Anticipation](https://mlsanity.com/paper/2205.14022/) (2022)
- [Measurement to Meaning: A Validity-Centered Framework for AI Evaluation](https://mlsanity.com/paper/2505.10573/) (2025)
- [Long-Term Anticipation of Activities with Cycle Consistency](https://mlsanity.com/paper/2009.01142/) (2020)
- [Uncertainty-Aware Anticipation of Activities](https://mlsanity.com/paper/1908.09540/) (2019)
- [Improving the Validity and Practical Usefulness of AI/ML Evaluations Using an Estimands Framework](https://mlsanity.com/paper/2406.10366/) (2024)
- [Action Anticipation at a Glimpse: To What Extent Can Multimodal Cues Replace Video?](https://mlsanity.com/paper/2512.02846/) (2025)

## Abstract (arXiv)

Benchmark scores license claims about the capabilities being evaluated. We audit the inference licensed by an evaluation protocol, rather than the predictive model alone. Using procedural action anticipation as a controlled case study, we study a broader evaluation-validity failure mode: a protocol can remain temporally causal and free of classical target leakage while still supplying a privileged intermediate representation. The failure is not merely optimistic accuracy, but a mismatch between the capability claimed and the construct actually measured. On Breakfast, matched recognizer-history and GT-history regimes score 30.3% and 62.3% (Delta_PH(R_BA) = +32.0, 95% Student-t interval [+24.1, +39.8]). A fixed-checkpoint 2 x 2 intervention isolates a +14.6-point test-time oracle contrast; a matched no-video provenance probe yields a +26.8-point gap; and a boundary-independent query/history control retains a +19.4-point gap. We refer to this discrepancy as a privileged-information gap, defined relative to a specified non-oracle recovery pipeline, and propose a reusable four-condition audit. Across five evaluation settings the gap is heterogeneous; a standard 50 Salads long-term-anticipation re-implementation provides cautious support beyond the audit-specific next-action construction.