# Algorithm Selection with Probing Trajectories: Benchmarking the Choice of Classifier Model

arXiv: https://arxiv.org/abs/2501.11414 · HTML: https://mlsanity.com/paper/2501.11414/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 20 Jan 2025.

## Builds on (most cited first, 22 of 22)

- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)
- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [HIVE-COTE 2.0: a new meta ensemble for time series classification](https://mlsanity.com/paper/2104.07551/) (2021)
- [Automated Algorithm Selection: Survey and Perspectives](https://mlsanity.com/paper/1811.11597/) (2018)
- [sktime: A Unified Interface for Machine Learning with Time Series](https://mlsanity.com/paper/1909.07872/) (2019)
- [A Time Series Forest for Classification and Feature Extraction](https://mlsanity.com/paper/1302.2277/) (2013)
- [shapeDTW: shape Dynamic Time Warping](https://mlsanity.com/paper/1606.01601/) (2016)
- [Per-run Algorithm Selection with Warm-starting using Trajectory-based Features](https://mlsanity.com/paper/2204.09483/) (2022)
- [Towards Feature-Based Performance Regression Using Trajectory Data](https://mlsanity.com/paper/2102.05370/) (2021)
- [Towards Explainable Exploratory Landscape Analysis: Extreme Feature Selection for Classifying BBOB Functions](https://mlsanity.com/paper/2102.00736/) (2021)
- [Automated Algorithm Selection: from Feature-Based to Feature-Free Approaches](https://mlsanity.com/paper/2203.13392/) (2022)
- [Deep Learning as a Competitive Feature-Free Approach for Automated Algorithm Selection on the Traveling Salesperson Problem](https://mlsanity.com/paper/2006.15968/) (2020)
- [Comparing Algorithm Selection Approaches on Black-Box Optimization Problems](https://mlsanity.com/paper/2306.17585/) (2023)
- [To Switch or not to Switch: Predicting the Benefit of Switching between Algorithms based on Trajectory Features](https://mlsanity.com/paper/2302.09075/) (2023)
- [Trajectory-based Algorithm Selection with Warm-starting](https://mlsanity.com/paper/2204.06397/) (2022)
- [DynamoRep: Trajectory-Based Population Dynamics for Classification of Black-box Optimization Problems](https://mlsanity.com/paper/2306.05438/) (2023)
- [On the Utility of Probing Trajectories for Algorithm-Selection](https://mlsanity.com/paper/2401.12745/) (2024)
- [MA-BBOB: Many-Affine Combinations of BBOB Functions for Evaluating AutoML Approaches in Noiseless Numerical Black-Box Optimization Contexts](https://mlsanity.com/paper/2306.10627/) (2023)
- [Improving Algorithm-Selection and Performance-Prediction via Learning Discriminating Training Samples](https://mlsanity.com/paper/2404.05359/) (2024)
- [Identifying Easy Instances to Improve Efficiency of ML Pipelines for Algorithm-Selection](https://mlsanity.com/paper/2406.16999/) (2024)

## Abstract (arXiv)

Recent approaches to training algorithm selectors in the black-box optimisation domain have advocated for the use of training data that is algorithm-centric in order to encapsulate information about how an algorithm performs on an instance, rather than relying on information derived from features of the instance itself. Probing-trajectories that consist of a sequence of objective performance per function evaluation obtained from a short run of an algorithm have recently shown particular promise in training accurate selectors. However, training models on this type of data requires an appropriately chosen classifier given the sequential nature of the data. There are currently no clear guidelines for choosing the most appropriate classifier for algorithm selection using time-series data from the plethora of models available. To address this, we conduct a large benchmark study using 17 different classifiers and three types of trajectory on a classification task using the BBOB benchmark suite using both leave-one-instance out and leave-one-problem out cross-validation. In contrast to previous studies using tabular data, we find that the choice of classifier has a significant impact, showing that feature-based and interval-based models are the best choices.