Partition Scores Are Not System Scores: Deployment-Fidelity Gaps in Decomposed Algorithm Selection
Not on the map. Why?
Abstract (arXiv)
Oracle-style quantities, including virtual best solvers, selected-portfolio VBS, virtual-best encodings, and best-in-family summaries, are widely reported as upper bounds on what a deployable selector could achieve. In decomposed algorithm selection, an analogous partition-level score grants an oracle choice of the best algorithm within the selected family; once the family selector is fixed, the deployable system must replace that within-family oracle with a learned within-family selector. We define the deployment-fidelity gap G(R) as the difference between partition-level and deployable end-to-end utility and derive two accounting consequences: a per-instance margin-regret stability condition that tells us when a partition-time family choice is deployment-optimal, and a sharp partition-only identification interval that, when it strictly crosses zero, prevents the partition-level report from certifying the deployable winner. Across five public algorithm-selection benchmarks spanning tabular AutoML and combinatorial CSP/SAT, every decomposed pipeline has positive G(R), ranging from 0.012 on TabZilla to 0.13 on PROTEUS-2014. Four of ten decomposed-versus-flat decisions have sign-changing point estimates; on PROTEUS-2014, a 33-point partition advantage shrinks to a 20-point end-to-end advantage. A training-side validation gap-correction diagnostic recovers the point-estimate deployable sign on all four sign-changing cells; it is a reporting aid, not a substitute for direct end-to-end evaluation. Partition and end-to-end scores should be reported side by side.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 23 references that are in the map. First posted on arXiv 12 Sep 2026.
Builds on
12 of 23 references.
References · 12
- 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
- 2025TabArena: A Living Benchmark for Machine Learning on Tabular Datacited by 176
- 2018Automated Algorithm Selection: Survey and Perspectivescited by 114
- 2024A Closer Look at Deep Learning Methods on Tabular Datasetscited by 41
- 2015ASlib: A Benchmark Library for Algorithm Selectioncited by 37
- 2023TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applicationscited by 33
- 2018The Algorithm Selection Competitions 2015 and 2017cited by 12
- 2021Algorithm Selection on a Meta Levelcited by 11
- 2023PS-AAS: Portfolio Selection for Automated Algorithm Selection in Black-Box Optimizationcited by 5
- 2013Proteus: A Hierarchical Portfolio of Solvers and Transformationscited by 4
- 2025The Pitfalls of Benchmarking in Algorithm Selection: What We Are Getting Wrongcited by 3
- 2023Learning to Select SAT Encodings for Pseudo-Boolean and Linear Integer Constraintscited by 2
Built on by
Not cited yet.