# Explainable Benchmarking for Iterative Optimization Heuristics

arXiv: https://arxiv.org/abs/2401.17842 · HTML: https://mlsanity.com/paper/2401.17842/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 80 references that are in the map.
- First posted on arXiv 31 Jan 2024.

## Builds on (most cited first, 28 of 28)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [An Easy-to-use Real-world Multi-objective Optimization Problem Suite](https://mlsanity.com/paper/2009.12867/) (2020)
- [Benchmarking in Optimization: Best Practice and Open Issues](https://mlsanity.com/paper/2007.03488/) (2020)
- [Automated Algorithm Selection on Continuous Black-Box Problems By Combining Exploratory Landscape Analysis and Machine Learning](https://mlsanity.com/paper/1711.08921/) (2017)
- [A Survey of Methods for Automated Algorithm Configuration](https://mlsanity.com/paper/2202.01651/) (2022)
- [BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters](https://mlsanity.com/paper/1908.06756/) (2019)
- [Covariance Matrix Adaptation MAP-Annealing](https://mlsanity.com/paper/2205.10752/) (2022)
- [IOHanalyzer: Detailed Performance Analyses for Iterative Optimization Heuristics](https://mlsanity.com/paper/2007.03953/) (2020)
- [Tuning as a Means of Assessing the Benefits of New Ideas in Interplay with Existing Algorithmic Modules](https://mlsanity.com/paper/2102.12905/) (2021)
- [DoE2Vec: Deep-learning Based Features for Exploratory Landscape Analysis](https://mlsanity.com/paper/2304.01219/) (2023)
- [Modular Differential Evolution](https://mlsanity.com/paper/2304.09524/) (2023)
- [Applying Genetic Programming to Improve Interpretability in Machine Learning Models](https://mlsanity.com/paper/2005.09512/) (2020)
- [BBOB Instance Analysis: Landscape Properties and Algorithm Performance across Problem Instances](https://mlsanity.com/paper/2211.16318/) (2022)
- [Structural bias in population-based algorithms](https://mlsanity.com/paper/1408.5350/) (2014)
- [Genetic Programming for Evolving a Front of Interpretable Models for Data Visualisation](https://mlsanity.com/paper/2001.09578/) (2020)
- [Identifying Properties of Real-World Optimisation Problems through a Questionnaire](https://mlsanity.com/paper/2011.05547/) (2020)
- [Explainable Landscape-Aware Optimization Performance Prediction](https://mlsanity.com/paper/2110.11633/) (2021)
- [The Importance of Landscape Features for Performance Prediction of Modular CMA-ES Variants](https://mlsanity.com/paper/2204.07431/) (2022)
- [Paradiseo: From a Modular Framework for Evolutionary Computation to the Automated Design of Metaheuristics ---22 Years of Paradiseo---](https://mlsanity.com/paper/2105.00420/) (2021)
- [Improving Nevergrad's Algorithm Selection Wizard NGOpt through Automated Algorithm Configuration](https://mlsanity.com/paper/2209.04412/) (2022)
- [Using Knowledge Graphs for Performance Prediction of Modular Optimization Algorithms](https://mlsanity.com/paper/2301.09876/) (2023)
- [Deep-BIAS: Detecting Structural Bias using Explainable AI](https://mlsanity.com/paper/2304.01869/) (2023)
- [Quantifying the Impact of Boundary Constraint Handling Methods on Differential Evolution](https://mlsanity.com/paper/2105.06757/) (2021)
- [The importance of being constrained: dealing with infeasible solutions in Differential Evolution and beyond](https://mlsanity.com/paper/2203.03512/) (2022)
- [Patterns of Convergence and Bound Constraint Violation in Differential Evolution on SBOX-COST Benchmarking Suite](https://mlsanity.com/paper/2305.12221/) (2023)

## Built on by (most cited first, 10 of 10)

- [LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristics](https://mlsanity.com/paper/2405.20132/) (2024)
- [BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristics](https://mlsanity.com/paper/2504.20183/) (2025)
- [A Survey of Features Used for Representing Black-box Single-objective Continuous Optimization](https://mlsanity.com/paper/2406.06629/) (2024)
- [Evolutionary Computation and Explainable AI: A Roadmap to Understandable Intelligent Systems](https://mlsanity.com/paper/2406.07811/) (2024)
- [Landscape-Aware Automated Algorithm Configuration using Multi-output Mixed Regression and Classification](https://mlsanity.com/paper/2409.01446/) (2024)
- [Tracing the Interactions of Modular CMA-ES Configurations Across Problem Landscapes](https://mlsanity.com/paper/2507.02331/) (2025)
- [From Performance to Understanding: A Vision for Explainable Automated Algorithm Design](https://mlsanity.com/paper/2511.16201/) (2025)
- [Quantifying the Impact of Modules and Their Interactions in the PSO-X Framework](https://mlsanity.com/paper/2601.04100/) (2026)
- [Pareto-Optimal Anytime Algorithms via Bayesian Racing](https://mlsanity.com/paper/2603.08493/) (2026)
- [Performance and Explainability Requirements of Evolutionary Algorithms in Real-World Physics-Informed Optimization](https://mlsanity.com/paper/2605.28164/) (2026)

## Abstract (arXiv)

Benchmarking heuristic algorithms is vital to understand under which conditions and on what kind of problems certain algorithms perform well. In most current research into heuristic optimization algorithms, only a very limited number of scenarios, algorithm configurations and hyper-parameter settings are explored, leading to incomplete and often biased insights and results. This paper presents a novel approach we call explainable benchmarking. Introducing the IOH-Xplainer software framework, for analyzing and understanding the performance of various optimization algorithms and the impact of their different components and hyper-parameters. We showcase the framework in the context of two modular optimization frameworks. Through this framework, we examine the impact of different algorithmic components and configurations, offering insights into their performance across diverse scenarios. We provide a systematic method for evaluating and interpreting the behaviour and efficiency of iterative optimization heuristics in a more transparent and comprehensible manner, allowing for better benchmarking and algorithm design.