# Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisons

arXiv: https://arxiv.org/abs/2608.07303 · HTML: https://mlsanity.com/paper/2608.07303/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 29 of 29)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [Do CIFAR-10 Classifiers Generalize to CIFAR-10?](https://mlsanity.com/paper/1806.00451/) (2018)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)
- [FLAML: A Fast and Lightweight AutoML Library](https://mlsanity.com/paper/1911.04706/) (2019)
- [Preserving Statistical Validity in Adaptive Data Analysis](https://mlsanity.com/paper/1411.2664/) (2014)
- [An Open Source AutoML Benchmark](https://mlsanity.com/paper/1907.00909/) (2019)
- [The Ladder: A Reliable Leaderboard for Machine Learning Competitions](https://mlsanity.com/paper/1502.04585/) (2015)
- [AMLB: an AutoML Benchmark](https://mlsanity.com/paper/2207.12560/) (2022)
- [Adaptive Memory Admission Control for LLM Agents](https://mlsanity.com/paper/2603.04549/) (2026)
- [KIS-S: A GPU-Aware Kubernetes Inference Simulator with RL-Based Auto-Scaling](https://mlsanity.com/paper/2507.07932/) (2025)
- [AMP4EC: Adaptive Model Partitioning Framework for Efficient Deep Learning Inference in Edge Computing Environments](https://mlsanity.com/paper/2504.00407/) (2025)
- [LLMs as Orchestrators: Constraint-Compliant Multi-Agent Optimization for Recommendation Systems](https://mlsanity.com/paper/2601.19121/) (2026)
- [Readout-Side Bypass for Residual Hybrid Quantum-Classical Models](https://mlsanity.com/paper/2511.20922/) (2025)
- [Adaptive GPU Resource Allocation for Multi-Agent Collaborative Reasoning in Serverless Environments](https://mlsanity.com/paper/2512.22149/) (2025)
- [Deployment-Time Memorization in Foundation-Model Agents](https://mlsanity.com/paper/2606.10062/) (2026)
- [AAPA: An Archetype-Aware Predictive Autoscaler with Uncertainty Quantification for Serverless Workloads on Kubernetes](https://mlsanity.com/paper/2507.05653/) (2025)
- [When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control](https://mlsanity.com/paper/2605.26418/) (2026)
- [RobustExplain: Evaluating Robustness of LLM-Based Explanation Agents for Recommendation](https://mlsanity.com/paper/2601.19120/) (2026)
- [Serverless GPU Architecture for Enterprise HR Analytics: A Production-Scale BDaaS Implementation](https://mlsanity.com/paper/2510.19689/) (2025)
- [CarbonEdge: Carbon-Aware Deep Learning Inference Framework for Sustainable Edge Computing](https://mlsanity.com/paper/2603.27420/) (2026)
- [Robust Explanations for User Trust in Enterprise NLP Systems](https://mlsanity.com/paper/2604.12069/) (2026)
- [EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms](https://mlsanity.com/paper/2606.04145/) (2026)

## Built on by (most cited first, 1 of 1)

- [Budget Boundary Effects in Test-Time Mathematical Reasoning](https://mlsanity.com/paper/2609.38699/) (2026)

## Abstract (arXiv)

Comparisons between AutoML systems at short time budgets -- tens of seconds rather than hours -- are common in tool READMEs and workshop papers, and they are easy to get wrong. We report a case study in which a simple AutoML engine, Orcetra, appeared to beat FLAML and AutoGluon on 513 OpenML datasets, winning 57.1% of them at a nominal 60-second budget and 78.4% of datasets against FLAML alone at 30 seconds. Both margins came from protocol defects that a results table cannot show. The search loop scored every candidate on the test split and reported the best, making the headline metric a maximum over dozens of noisy estimates while the baselines selected on training data and touched the test set once; and the budget was checked before launching a candidate but never enforced during one, so the system consumed a median of 120 s against a 60-second budget, 2.24x the wall-clock AutoGluon used. Re-running with selection moved to a validation split, the deadline enforced externally and every framework pinned to an equal share of the machine, Orcetra's win rate on the re-run subset falls from 59.4% to 34.3% and no pairwise difference against either competitor remains significant. Recording both estimands inside a single search lets us attribute the collapse: the selection rule accounts for 4.8 percentage points and unequal compute for most of the rest. The same traces give the selection bias as a function of budget, measured rather than assumed: it grows with K but reaches only 0.27 accuracy points, about five times below the σ√2ln K bound a marginal-standard-error argument predicts, because candidates scored on shared test rows cancel most of the noise. We close with a checklist for short-budget comparisons. Code, per-dataset results and the scripts that regenerate every number and figure in the paper are released with it.