# When 10,000 Windows Are Not 10,000 Tests: Auditing Statistical Confidence in Sliding-Window Time-Series Classification

arXiv: https://arxiv.org/abs/2609.30721 · HTML: https://mlsanity.com/paper/2609.30721/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Builds on (most cited first, 9 of 9)

- [MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification](https://mlsanity.com/paper/2012.08791/) (2020)
- [Using simulation studies to evaluate statistical methods](https://mlsanity.com/paper/1712.03198/) (2017)
- [Human Activity Recognition Based on Wearable Sensor Data: A Standardization of the State-of-the-Art](https://mlsanity.com/paper/1806.05226/) (2018)
- [Improving the Validity and Practical Usefulness of AI/ML Evaluations Using an Estimands Framework](https://mlsanity.com/paper/2406.10366/) (2024)
- [Subject Cross Validation in Human Activity Recognition](https://mlsanity.com/paper/1904.02666/) (2019)
- [Application of the hierarchical bootstrap to multi-level data in neuroscience](https://mlsanity.com/paper/2007.07797/) (2020)
- [Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data](https://mlsanity.com/paper/2606.26422/) (2026)
- [Beyond Point Estimates: Reliable Evaluation of Prediction Performance Metrics under Clustered Data](https://mlsanity.com/paper/2606.03656/) (2026)
- [Disjoint or Overlapping? Inference Windowing for Reconstruction-Based Time Series Anomaly Detection](https://mlsanity.com/paper/2606.09874/) (2026)

## Abstract (arXiv)

Sliding-window classifiers are often evaluated on thousands of overlapping test windows, even though neighboring predictions share observations and remain nested within recordings and subjects. Subject-disjoint evaluation prevents one form of leakage but does not make those test windows independent. We present a practical audit that maps three claims - performance on observed recordings, future recordings from observed subjects, and unseen subjects - to explicit aggregation rules and established dependence-robust inference. At 75% overlap, controlled simulations give 16.9% Type-I error for IID observed-record inference and 7.2% for session-centered Bartlett-HAC: a substantial improvement with residual miscalibration. Audits of frozen WISDM and HARTH predictions show that nearly fourfold growth in test rows yields only 1.75-1.94-fold variance-equivalent information growth. At that overlap, fixed-record paired Accuracy-difference intervals are 1.22-1.66 times the IID widths; this inflation is not universal at zero overlap. On HARTH, paired Accuracy-difference intervals include zero across three overlap settings, whereas Macro-F1 favors MiniROCKET. Independent recomputation, common-session checks, class-level results, and separately seeded calibration make the audit's scope and limitations inspectable. The resulting workflow distinguishes additional predictions from additional independent evidence.