# Partition Scores Are Not System Scores: Deployment-Fidelity Gaps in Decomposed Algorithm Selection

arXiv: https://arxiv.org/abs/2609.13785 · HTML: https://mlsanity.com/paper/2609.13785/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 12 Sep 2026.

## Builds on (most cited first, 12 of 12)

- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [Automated Algorithm Selection: Survey and Perspectives](https://mlsanity.com/paper/1811.11597/) (2018)
- [A Closer Look at Deep Learning Methods on Tabular Datasets](https://mlsanity.com/paper/2407.00956/) (2024)
- [ASlib: A Benchmark Library for Algorithm Selection](https://mlsanity.com/paper/1506.02465/) (2015)
- [TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications](https://mlsanity.com/paper/2311.02971/) (2023)
- [The Algorithm Selection Competitions 2015 and 2017](https://mlsanity.com/paper/1805.01214/) (2018)
- [Algorithm Selection on a Meta Level](https://mlsanity.com/paper/2107.09414/) (2021)
- [PS-AAS: Portfolio Selection for Automated Algorithm Selection in Black-Box Optimization](https://mlsanity.com/paper/2310.10685/) (2023)
- [Proteus: A Hierarchical Portfolio of Solvers and Transformations](https://mlsanity.com/paper/1306.5606/) (2013)
- [The Pitfalls of Benchmarking in Algorithm Selection: What We Are Getting Wrong](https://mlsanity.com/paper/2505.07750/) (2025)
- [Learning to Select SAT Encodings for Pseudo-Boolean and Linear Integer Constraints](https://mlsanity.com/paper/2307.09342/) (2023)

## Abstract (arXiv)

Oracle-style quantities, including virtual best solvers, selected-portfolio VBS, virtual-best encodings, and best-in-family summaries, are widely reported as upper bounds on what a deployable selector could achieve. In decomposed algorithm selection, an analogous partition-level score grants an oracle choice of the best algorithm within the selected family; once the family selector is fixed, the deployable system must replace that within-family oracle with a learned within-family selector. We define the deployment-fidelity gap G(R) as the difference between partition-level and deployable end-to-end utility and derive two accounting consequences: a per-instance margin-regret stability condition that tells us when a partition-time family choice is deployment-optimal, and a sharp partition-only identification interval that, when it strictly crosses zero, prevents the partition-level report from certifying the deployable winner. Across five public algorithm-selection benchmarks spanning tabular AutoML and combinatorial CSP/SAT, every decomposed pipeline has positive G(R), ranging from 0.012 on TabZilla to 0.13 on PROTEUS-2014. Four of ten decomposed-versus-flat decisions have sign-changing point estimates; on PROTEUS-2014, a 33-point partition advantage shrinks to a 20-point end-to-end advantage. A training-side validation gap-correction diagnostic recovers the point-estimate deployable sign on all four sign-changing cells; it is a reporting aid, not a substitute for direct end-to-end evaluation. Partition and end-to-end scores should be reported side by side.