# Evaluating Model Retraining under Drift: Paired Comparisons of Cumulative Subgroup Disparity

arXiv: https://arxiv.org/abs/2609.09788 · HTML: https://mlsanity.com/paper/2609.09788/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 9 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Delayed Impact of Fair Machine Learning](https://mlsanity.com/paper/1803.04383/) (2018)
- [Runaway Feedback Loops in Predictive Policing](https://mlsanity.com/paper/1706.09847/) (2017)
- [FAHT: An Adaptive Fairness-aware Decision Tree Classifier](https://mlsanity.com/paper/1907.07237/) (2019)
- [Supervised Algorithmic Fairness in Distribution Shifts: A Survey](https://mlsanity.com/paper/2402.01327/) (2024)
- [Runtime Monitoring of Dynamic Fairness Properties](https://mlsanity.com/paper/2305.04699/) (2023)
- [Preventing Discriminatory Decision-making in Evolving Data Streams](https://mlsanity.com/paper/2302.08017/) (2023)
- [When to retrain a machine learning model](https://mlsanity.com/paper/2505.14903/) (2025)
- [Stream-Based Monitoring of Algorithmic Fairness](https://mlsanity.com/paper/2501.18331/) (2025)
- [Is it Still Fair? A Comparative Evaluation of Fairness Algorithms through the Lens of Covariate Drift](https://mlsanity.com/paper/2409.12428/) (2024)
- [RCCDA: Adaptive Model Updates in the Presence of Concept Drift under a Constrained Resource Budget](https://mlsanity.com/paper/2505.24149/) (2025)
- [Auditing Fairness under Model Updates: Fundamental Complexity and Property-Preserving Updates](https://mlsanity.com/paper/2601.05909/) (2026)
- [When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints](https://mlsanity.com/paper/2608.19488/) (2026)

## Abstract (arXiv)

Choosing when to retrain a deployed classifier requires assessing subgroup error rates across the sequence of models used, including periods between updates. We compare complete scheduled, loss-triggered, and subgroup-gap-triggered policies with retaining the initial model on the same observations and delayed labels. For true-positive and false-positive rates separately, the outcome is the paired difference in absolute subgroup gaps summed over deployment windows. Population evaluation in simulation, action records, and alternative schedules assess how measurement and retraining behaviour affect these comparisons. In a follow-up sample of 400 new trajectories per condition across two simulated drift regimes, all three policies had lower mean cumulative disparity, equivalent to reductions of 0.04 to 0.88 percentage points in the average gap per window. Evaluating the unchanged models against the known generating distributions preserved all mean directions, but finite-window and population comparisons agreed on whether updating increased, reduced or left cumulative disparity unchanged in 69 to 92 percent of trajectories. Under subgroup-specific drift, smaller true-positive-rate gaps accompanied lower sensitivity in both groups. In an exploratory American Community Survey replay, person weighting reversed all three race false-positive-rate mean comparisons without changing predictions or actions; all three weighted intervals included zero. Policy comparisons require group-specific rates, action distributions, and an explicit evaluation population alongside mean disparity. These analyses are non-confirmatory. Shared replay requires policy-independent observations and complete labels after the specified delay.