# Sensitivity Analysis of RF+clust for Leave-one-problem-out Performance Prediction

arXiv: https://arxiv.org/abs/2305.19375 · HTML: https://mlsanity.com/paper/2305.19375/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 10 of 10)

- [A Random Forest Guided Tour](https://mlsanity.com/paper/1511.05741/) (2015)
- [Automated Algorithm Selection: Survey and Perspectives](https://mlsanity.com/paper/1811.11597/) (2018)
- [Automated Algorithm Selection on Continuous Black-Box Problems By Combining Exploratory Landscape Analysis and Machine Learning](https://mlsanity.com/paper/1711.08921/) (2017)
- [Per-run Algorithm Selection with Warm-starting using Trajectory-based Features](https://mlsanity.com/paper/2204.09483/) (2022)
- [Landscape-Aware Fixed-Budget Performance Regression and Algorithm Selection for Modular CMA-ES Variants](https://mlsanity.com/paper/2006.09855/) (2020)
- [Towards Feature-Based Performance Regression Using Trajectory Data](https://mlsanity.com/paper/2102.05370/) (2021)
- [The Impact of Hyper-Parameter Tuning for Landscape-Aware Performance Regression and Algorithm Selection](https://mlsanity.com/paper/2104.09272/) (2021)
- [Hyper-Process Model: A Zero-Shot Learning algorithm for Regression Problems based on Shape Analysis](https://mlsanity.com/paper/1810.10330/) (2018)
- [Explainable Landscape Analysis in Automated Algorithm Performance Prediction](https://mlsanity.com/paper/2203.11828/) (2022)
- [RF+clust for Leave-One-Problem-Out Performance Prediction](https://mlsanity.com/paper/2301.09524/) (2023)

## Built on by (most cited first, 2 of 2)

- [A Survey of Features Used for Representing Black-box Single-objective Continuous Optimization](https://mlsanity.com/paper/2406.06629/) (2024)
- [carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks](https://mlsanity.com/paper/2506.06143/) (2025)

## Abstract (arXiv)

Leave-one-problem-out (LOPO) performance prediction requires machine learning (ML) models to extrapolate algorithms' performance from a set of training problems to a previously unseen problem. LOPO is a very challenging task even for state-of-the-art approaches. Models that work well in the easier leave-one-instance-out scenario often fail to generalize well to the LOPO setting. To address the LOPO problem, recent work suggested enriching standard random forest (RF) performance regression models with a weighted average of algorithms' performance on training problems that are considered similar to a test problem. More precisely, in this RF+clust approach, the weights are chosen proportionally to the distances of the problems in some feature space. Here in this work, we extend the RF+clust approach by adjusting the distance-based weights with the importance of the features for performance regression. That is, instead of considering cosine distance in the feature space, we consider a weighted distance measure, with weights depending on the relevance of the feature for the regression model. Our empirical evaluation of the modified RF+clust approach on the CEC 2014 benchmark suite confirms its advantages over the naive distance measure. However, we also observe room for improvement, in particular with respect to more expressive feature portfolios.