# Candidate Comparability Before Promotion: Conditional Validation in Adaptive Network Intrusion Detection

arXiv: https://arxiv.org/abs/2609.04388 · HTML: https://mlsanity.com/paper/2609.04388/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 92 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 30 of 36)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Supervised learning with quantum enhanced feature spaces](https://mlsanity.com/paper/1804.11326/) (2018)
- [Learning under Concept Drift: A Review](https://mlsanity.com/paper/2004.05785/) (2020)
- [Power of data in quantum machine learning](https://mlsanity.com/paper/2011.01938/) (2020)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Time-uniform, nonparametric, nonasymptotic confidence sequences](https://mlsanity.com/paper/1810.08240/) (2018)
- [Estimating means of bounded random variables by betting](https://mlsanity.com/paper/2010.09686/) (2020)
- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Machine Learning Operations (MLOps): Overview, Definition, and Architecture](https://mlsanity.com/paper/2205.02302/) (2022)
- [A Survey on Negative Transfer](https://mlsanity.com/paper/2009.00909/) (2020)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Predicting with Confidence on Unseen Distributions](https://mlsanity.com/paper/2107.03315/) (2021)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [Are Labels Always Necessary for Classifier Accuracy Evaluation?](https://mlsanity.com/paper/2007.02915/) (2020)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift](https://mlsanity.com/paper/2206.13089/) (2022)
- [Exponential concentration in quantum kernel methods](https://mlsanity.com/paper/2208.11060/) (2022)
- [From Concept Drift to Model Degradation: An Overview on Performance-Aware Drift Detectors](https://mlsanity.com/paper/2203.11070/) (2022)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [Transcending Transcend: Revisiting Malware Classification in the Presence of Concept Drift](https://mlsanity.com/paper/2010.03856/) (2020)
- [Reactive Soft Prototype Computing for Concept Drift Streams](https://mlsanity.com/paper/2007.05432/) (2020)
- [Continuous Learning for Android Malware Detection](https://mlsanity.com/paper/2302.04332/) (2023)
- [Active learning for data streams: a survey](https://mlsanity.com/paper/2302.08893/) (2023)
- [Quantum Kernel Methods under Scrutiny: A Benchmarking Study](https://mlsanity.com/paper/2409.04406/) (2024)
- [Online Active Model Selection for Pre-trained Classifiers](https://mlsanity.com/paper/2010.09818/) (2020)
- [(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancy](https://mlsanity.com/paper/2306.00312/) (2023)
- [SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection](https://mlsanity.com/paper/2305.00550/) (2023)
- [Model Assessment and Selection under Temporal Distribution Shift](https://mlsanity.com/paper/2402.08672/) (2024)
- [A comprehensive analysis of concept drift locality in data streams](https://mlsanity.com/paper/2311.06396/) (2023)
- [All models are wrong, some are useful: Model Selection with Limited Labels](https://mlsanity.com/paper/2410.13609/) (2024)

## Abstract (arXiv)

Adaptive network intrusion detection systems retrain classifiers after drift alarms, but an alarm detects change; it does not establish that a challenger should replace the deployed incumbent. Promotion is security-relevant because it changes the model responsible for subsequent attack detection, and evaluating it has a methodological problem: promotion conclusions may depend on how the challenger was constructed and on how much evidence supports it. We test that dependence on CICIDS2017, UNSW-NB15 and ToN-IoT with self-contained challenger pipelines, nested candidate-size controls, a common-harness comparison of nine update policies, and a final sensitivity confining every exact feature vector to one evaluation, training or probe role. Incumbent-owned frozen preprocessing amplified apparent promotion harm; with self-contained challenger pipelines the mean full-drift harm did not persist. Raising nominal candidate evidence from 512 to 2,000 samples per class improved promotion under pool-constructed progressive drift by +0.53, +1.67 and +0.38 balanced-accuracy points: positive and statistically resolved in all three benchmarks, but materially benchmark-dependent rather than homogeneous, and driven mainly by fewer false positives. Policy conclusions were partially robust: policy ordering changed with candidate comparability, no policy globally dominated, and earlier compatibility statements for a label-free estimator and a calibrated ensemble narrowed. Validation helped evidence-disadvantaged challengers but added no average benefit at parity. Thirteen replays on real, time-ordered traffic showed no net harm from always deploying. Challenger construction and evidence should be controlled, reported and interpreted explicitly when promotion is evaluated.