# Model Assessment and Selection under Temporal Distribution Shift

arXiv: https://arxiv.org/abs/2402.08672 · HTML: https://mlsanity.com/paper/2402.08672/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 13 Feb 2024.

## Builds on (most cited first, 16 of 16)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [Non-stationary Stochastic Optimization](https://mlsanity.com/paper/1307.5449/) (2013)
- [lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits](https://mlsanity.com/paper/1312.7308/) (2013)
- [Strongly Adaptive Online Learning](https://mlsanity.com/paper/1502.07073/) (2015)
- [Corralling a Band of Bandit Algorithms](https://mlsanity.com/paper/1612.06246/) (2016)
- [Online Optimization : Competing with Dynamic Comparators](https://mlsanity.com/paper/1501.06225/) (2015)
- [Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approach](https://mlsanity.com/paper/2102.05406/) (2021)
- [Online Model Selection for Reinforcement Learning with Function Approximation](https://mlsanity.com/paper/2011.09750/) (2020)
- [Adapting to Online Label Shift with Provable Guarantees](https://mlsanity.com/paper/2207.02121/) (2022)
- [Sequential Nonparametric Testing with the Law of the Iterated Logarithm](https://mlsanity.com/paper/1506.03486/) (2015)
- [An Adaptive Algorithm for Learning with Unknown Distribution Drift](https://mlsanity.com/paper/2305.02252/) (2023)
- [Multiple Change-point Detection: a Selective Overview](https://mlsanity.com/paper/1512.04093/) (2015)
- [Learning with a Drifting Target Concept](https://mlsanity.com/paper/1505.05215/) (2015)
- [A Stability Principle for Learning under Non-Stationarity](https://mlsanity.com/paper/2310.18304/) (2023)
- [An Adaptive Method for Weak Supervision with Drifting Data](https://mlsanity.com/paper/2306.01658/) (2023)

## Built on by (most cited first, 12 of 12)

- [Learning Robust Spectral Dynamics for Temporal Domain Generalization](https://mlsanity.com/paper/2505.12585/) (2025)
- [Adaptive Estimation and Learning under Temporal Distribution Shift](https://mlsanity.com/paper/2505.15803/) (2025)
- [Jacobian-Velocity Bounds for Deployment Risk Under Covariate Drift](https://mlsanity.com/paper/2605.04932/) (2026)
- [Distribution-Free Predictive Inference under Unknown Temporal Drift](https://mlsanity.com/paper/2406.06516/) (2024)
- [Adaptive Window Selection for Financial Risk Forecasting](https://mlsanity.com/paper/2603.01157/) (2026)
- [Efficiently Learning Drifting Halfspaces with Massart Noise](https://mlsanity.com/paper/2606.11149/) (2026)
- [BP-TTA: Balanced and Prototype-Guided Test-Time Adaptation in Dynamic Scenarios](https://mlsanity.com/paper/2606.31420/) (2026)
- [FreKoo++: Learning Continuous Spectral Dynamics for Temporal Domain Generalization](https://mlsanity.com/paper/2608.22224/) (2026)
- [Candidate Comparability Before Promotion: Conditional Validation in Adaptive Network Intrusion Detection](https://mlsanity.com/paper/2609.04388/) (2026)
- [Evidence-Gated Research: Statistically Controlled Model Adoption in Adaptive Search](https://mlsanity.com/paper/2610.01751/) (2026)
- [Adaptive Spectral-Koopman Dynamics Modeling for Temporal Domain Generalization](https://mlsanity.com/paper/2610.02822/) (2026)
- [LLM Evaluation is a Forecasting Problem](https://mlsanity.com/paper/2610.17018/) (2026)

## Abstract (arXiv)

We investigate model assessment and selection in a changing environment, by synthesizing datasets from both the current time period and historical epochs. To tackle unknown and potentially arbitrary temporal distribution shift, we develop an adaptive rolling window approach to estimate the generalization error of a given model. This strategy also facilitates the comparison between any two candidate models by estimating the difference of their generalization errors. We further integrate pairwise comparisons into a single-elimination tournament, achieving near-optimal model selection from a collection of candidates. Theoretical analyses and numerical experiments demonstrate the adaptivity of our proposed methods to the non-stationarity in data.