# Sequential algorithmic modification with test data reuse

arXiv: https://arxiv.org/abs/2203.11377 · HTML: https://mlsanity.com/paper/2203.11377/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 46 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 12 of 12)

- [The Ladder: A Reliable Leaderboard for Machine Learning Competitions](https://mlsanity.com/paper/1502.04585/) (2015)
- [Model Similarity Mitigates Test Set Overuse](https://mlsanity.com/paper/1905.12580/) (2019)
- [Calibrating Noise to Variance in Adaptive Data Analysis](https://mlsanity.com/paper/1712.07196/) (2017)
- [Max-Information, Differential Privacy, and Post-Selection Hypothesis Testing](https://mlsanity.com/paper/1604.03924/) (2016)
- [Generalization for Adaptively-chosen Estimators via Stable Median](https://mlsanity.com/paper/1706.05069/) (2017)
- [Adaptive Learning with Robust Generalization Guarantees](https://mlsanity.com/paper/1602.07726/) (2016)
- [SAFFRON: an adaptive algorithm for online control of the false discovery rate](https://mlsanity.com/paper/1802.09098/) (2018)
- [Guaranteed Validity for Empirical Approaches to Adaptive Data Analysis](https://mlsanity.com/paper/1906.09231/) (2019)
- [Private False Discovery Rate Control](https://mlsanity.com/paper/1511.03803/) (2015)
- [Natural Analysts in Adaptive Data Analysis](https://mlsanity.com/paper/1901.11143/) (2019)
- [Approval policies for modifications to Machine Learning-Based Software as a Medical Device: A study of bio-creep](https://mlsanity.com/paper/1912.12413/) (2019)
- [Graphical approaches for the control of generalised error rates](https://mlsanity.com/paper/2004.01759/) (2020)

## Abstract (arXiv)

After initial release of a machine learning algorithm, the model can be fine-tuned by retraining on subsequently gathered data, adding newly discovered features, or more. Each modification introduces a risk of deteriorating performance and must be validated on a test dataset. It may not always be practical to assemble a new dataset for testing each modification, especially when most modifications are minor or are implemented in rapid succession. Recent works have shown how one can repeatedly test modifications on the same dataset and protect against overfitting by (i) discretizing test results along a grid and (ii) applying a Bonferroni correction to adjust for the total number of modifications considered by an adaptive developer. However, the standard Bonferroni correction is overly conservative when most modifications are beneficial and/or highly correlated. This work investigates more powerful approaches using alpha-recycling and sequentially-rejective graphical procedures (SRGPs). We introduce novel extensions that account for correlation between adaptively chosen algorithmic modifications. In empirical analyses, the SRGPs control the error rate of approving unacceptable modifications and approve a substantially higher number of beneficial modifications than previous approaches.