# IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics

arXiv: https://arxiv.org/abs/1810.05281 · HTML: https://mlsanity.com/paper/1810.05281/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2018.

## Built on by (most cited first, 28 of 28)

- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristics](https://mlsanity.com/paper/2405.20132/) (2024)
- [Automated Dynamic Algorithm Configuration](https://mlsanity.com/paper/2205.13881/) (2022)
- [MetaBox: A Benchmark Platform for Meta-Black-Box Optimization with Reinforcement Learning](https://mlsanity.com/paper/2310.08252/) (2023)
- [DACBench: A Benchmark Library for Dynamic Algorithm Configuration](https://mlsanity.com/paper/2105.08541/) (2021)
- [Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimization](https://mlsanity.com/paper/2411.00625/) (2024)
- [Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOB](https://mlsanity.com/paper/2303.00890/) (2023)
- [OptunaHub: A Platform for Black-Box Optimization](https://mlsanity.com/paper/2510.02798/) (2025)
- [Meta Learning Black-Box Population-Based Optimizers](https://mlsanity.com/paper/2103.03526/) (2021)
- [EXPObench: Benchmarking Surrogate-based Optimisation Algorithms on Expensive Black-box Functions](https://mlsanity.com/paper/2106.04618/) (2021)
- [Hyperparameter Importance Analysis for Multi-Objective AutoML](https://mlsanity.com/paper/2405.07640/) (2024)
- [MetaBox-v2: A Unified Benchmark Platform for Meta-Black-Box Optimization](https://mlsanity.com/paper/2505.17745/) (2025)
- [MA-BBOB: Many-Affine Combinations of BBOB Functions for Evaluating AutoML Approaches in Noiseless Numerical Black-Box Optimization Contexts](https://mlsanity.com/paper/2306.10627/) (2023)
- [Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systems](https://mlsanity.com/paper/2505.15741/) (2025)
- [Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym](https://mlsanity.com/paper/2310.00077/) (2023)
- [Automated Algorithm Selection for Radar Network Configuration](https://mlsanity.com/paper/2205.03670/) (2022)
- [Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization](https://mlsanity.com/paper/2505.21321/) (2025)
- [Landscape-Aware Automated Algorithm Configuration using Multi-output Mixed Regression and Classification](https://mlsanity.com/paper/2409.01446/) (2024)
- [Log-normal Mutations and their Use in Detecting Surreptitious Fake Images](https://mlsanity.com/paper/2409.15119/) (2024)
- [LLaMEA-SAGE: Guiding Automated Algorithm Design with Structural Feedback from Explainable AI](https://mlsanity.com/paper/2601.21511/) (2026)
- [Quantum-Enhanced Selection Operators for Evolutionary Algorithms](https://mlsanity.com/paper/2206.10743/) (2022)
- [Explainable Model-specific Algorithm Selection for Multi-Label Classification](https://mlsanity.com/paper/2211.11227/) (2022)
- [OPTION: OPTImization Algorithm Benchmarking ONtology](https://mlsanity.com/paper/2211.11332/) (2022)
- [Illuminating the Diversity-Fitness Trade-Off in Black-Box Optimization](https://mlsanity.com/paper/2408.16393/) (2024)
- [Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Network](https://mlsanity.com/paper/2512.16491/) (2025)
- [Quantifying the Impact of Modules and Their Interactions in the PSO-X Framework](https://mlsanity.com/paper/2601.04100/) (2026)
- [Evaluating Real-World Generalizability of Algorithm Selection Models](https://mlsanity.com/paper/2606.02016/) (2026)
- [A practical DIRECT-type algorithm for medium-scale black-box global optimization](https://mlsanity.com/paper/2609.09796/) (2026)

## Abstract (arXiv)

IOHprofiler is a new tool for analyzing and comparing iterative optimization heuristics. Given as input algorithms and problems written in C or Python, it provides as output a statistical evaluation of the algorithms' performance by means of the distribution on the fixed-target running time and the fixed-budget function values. In addition, IOHprofiler also allows to track the evolution of algorithm parameters, making our tool particularly useful for the analysis, comparison, and design of (self-)adaptive algorithms. IOHprofiler is a ready-to-use software. It consists of two parts: an experimental part, which generates the running time data, and a post-processing part, which produces the summarizing comparisons and statistical evaluations. The experimental part is build on the COCO software, which has been adjusted to cope with optimization problems that are formulated as functions f:Sn → R with S being a discrete alphabet of integers. The post-processing part is our own work. It can be used as a stand-alone tool for the evaluation of running time data of arbitrary benchmark problems. It accepts as input files not only the output files of IOHprofiler, but also original COCO data files. The post-processing tool is designed for an interactive evaluation, allowing the user to chose the ranges and the precision of the displayed data according to his/her needs. IOHprofiler is available on GitHub at urlhttps://github.com/IOHprofiler.