# Evaluating Useful Surrogate Models for Configuration Tuning Beyond Accuracy: A Fitness Landscape Analysis Perspective

arXiv: https://arxiv.org/abs/2509.21945 · HTML: https://mlsanity.com/paper/2509.21945/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 133 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Builds on (most cited first, 30 of 46)

- [A Tutorial on Principal Component Analysis](https://mlsanity.com/paper/1404.1100/) (2014)
- [ParamILS: An Automatic Algorithm Configuration Framework](https://mlsanity.com/paper/1401.3492/) (2014)
- [The Impact of Automated Parameter Optimization on Defect Prediction Models](https://mlsanity.com/paper/1801.10270/) (2018)
- [Finding Faster Configurations using FLASH](https://mlsanity.com/paper/1801.02175/) (2018)
- [Multi-Objectivizing Software Configuration Tuning (for a single performance concern)](https://mlsanity.com/paper/2106.01331/) (2021)
- [BestConfig: Tapping the Performance Potential of Systems via Automatic Configuration Tuning](https://mlsanity.com/paper/1710.03439/) (2017)
- [Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation](https://mlsanity.com/paper/2110.12654/) (2021)
- [Do Performance Aspirations Matter for Guiding Software Configuration Tuning?](https://mlsanity.com/paper/2301.03290/) (2023)
- [Transfer Learning for Performance Modeling of Configurable Systems: An Exploratory Analysis](https://mlsanity.com/paper/1709.02280/) (2017)
- [MMO: Meta Multi-Objectivization for Software Configuration Tuning](https://mlsanity.com/paper/2112.07303/) (2021)
- [Greening Large Language Models of Code](https://mlsanity.com/paper/2309.04076/) (2023)
- [FEMOSAA: Feature Guided and Knee Driven Multi-Objective Optimization for Self-Adaptive Software](https://mlsanity.com/paper/1608.08933/) (2016)
- [An Uncertainty-Aware Approach to Optimal Configuration of Stream Processing Systems](https://mlsanity.com/paper/1606.06543/) (2016)
- [How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? A Critical Review and Methodological Guidance](https://mlsanity.com/paper/2002.09040/) (2020)
- [The Weights can be Harmful: Pareto Search versus Weighted Search in Multi-Objective Search-Based Software Engineering](https://mlsanity.com/paper/2202.03728/) (2022)
- [Predicting Configuration Performance in Multiple Environments with Sequential Meta-learning](https://mlsanity.com/paper/2402.03183/) (2024)
- [Predicting Software Performance with Divide-and-Learn](https://mlsanity.com/paper/2306.06651/) (2023)
- [Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimization](https://mlsanity.com/paper/2306.09803/) (2023)
- [LlamaTune: Sample-Efficient DBMS Configuration Tuning](https://mlsanity.com/paper/2203.05128/) (2022)
- [Hyperparameter Optimization for Effort Estimation](https://mlsanity.com/paper/1805.00336/) (2018)
- [Adapting Multi-objectivized Software Configuration Tuning](https://mlsanity.com/paper/2404.04744/) (2024)
- [Planning Landscape Analysis for Self-Adaptive Systems](https://mlsanity.com/paper/2203.12472/) (2022)
- [Distilled Lifelong Self-Adaptation for Configurable Systems](https://mlsanity.com/paper/2501.00840/) (2025)
- [PromiseTune: Unveiling Causally Promising and Explainable Configuration Tuning](https://mlsanity.com/paper/2507.05995/) (2025)
- [White-Box Analysis over Machine Learning: Modeling Performance of Configurable Systems](https://mlsanity.com/paper/2101.05362/) (2021)
- [Does Configuration Encoding Matter in Learning Software Performance? An Empirical Study on Encoding Schemes](https://mlsanity.com/paper/2203.15988/) (2022)
- [Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning](https://mlsanity.com/paper/2501.01876/) (2025)
- [Faster Configuration Performance Bug Testing with Neural Dual-level Prioritization](https://mlsanity.com/paper/2501.15392/) (2025)
- ["Sampling"' as a Baseline Optimizer for Search-based Software Engineering](https://mlsanity.com/paper/1608.07617/) (2016)
- [Using Bad Learners to find Good Configurations](https://mlsanity.com/paper/1702.05701/) (2017)

## Built on by (most cited first, 1 of 1)

- [Configuring RAG Systems: What Breaks, Where Originates, and How to Fix](https://mlsanity.com/paper/2610.18567/) (2026)

## Abstract (arXiv)

To efficiently tune configuration for better software system performance (e.g., latency) at the deployment and maintenance stage, many tuners have leveraged a surrogate model to expedite the process instead of solely relying on the profoundly expensive system measurement. As such, it is naturally believed that we need more accurate models. However, the fact of "accuracy can lie"-a somewhat surprising finding from prior work-has left us many unanswered questions regarding what role the surrogate model plays in configuration tuning. This paper provides the very first systematic exploration and discussion, together with a resolution proposal, to disclose the many faces of useful surrogate models for configuration tuning beyond accuracy, through the novel perspective of fitness landscape analysis. We present a theory as an alternative to accuracy for assessing the model usefulness in tuning, based on which we conduct an extensive empirical study involving up to 27,000 cases. Drawing on the above, we propose Model4Tune, an automated predictive tool that estimates which model-tuner pairs are the best for an unforeseen system without expensive tuner profiling. Our results suggest that Model4Tune, as one of the first of its kind, performs significantly better than random guessing in 79%-82% of the cases, hence greatly mitigating the required efforts in engineering configuration for software systems. Our results not only shed light on the possible future research directions but also offer a practical resolution that can assist practitioners in evaluating the most useful model for configuration tuning.