# Benchmarking in Optimization: Best Practice and Open Issues

arXiv: https://arxiv.org/abs/2007.03488 · HTML: https://mlsanity.com/paper/2007.03488/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2020.

## Built on by (most cited first, 30 of 38)

- [BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices](https://mlsanity.com/paper/2411.12990/) (2024)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [Deconstructing What Makes a Good Optimizer for Language Models](https://mlsanity.com/paper/2407.07972/) (2024)
- [Benchopt: Reproducible, efficient and collaborative optimization benchmarks](https://mlsanity.com/paper/2206.13424/) (2022)
- [Reproducibility in Evolutionary Computation](https://mlsanity.com/paper/2102.03380/) (2021)
- [Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture](https://mlsanity.com/paper/2411.13768/) (2024)
- [Explainable Benchmarking for Iterative Optimization Heuristics](https://mlsanity.com/paper/2401.17842/) (2024)
- [BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristics](https://mlsanity.com/paper/2504.20183/) (2025)
- [A Survey on Learnable Evolutionary Algorithms for Scalable Multiobjective Optimization](https://mlsanity.com/paper/2206.11526/) (2022)
- [Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systems](https://mlsanity.com/paper/2505.15741/) (2025)
- [The Importance of Landscape Features for Performance Prediction of Modular CMA-ES Variants](https://mlsanity.com/paper/2204.07431/) (2022)
- [Using Automated Algorithm Configuration for Parameter Control](https://mlsanity.com/paper/2302.12334/) (2023)
- [A Framework for Generating Informative Benchmark Instances](https://mlsanity.com/paper/2205.14753/) (2022)
- [Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym](https://mlsanity.com/paper/2310.00077/) (2023)
- [Towards Standardising Reinforcement Learning Approaches for Production Scheduling Problems](https://mlsanity.com/paper/2104.08196/) (2021)
- [Tools for Landscape Analysis of Optimisation Problems in Procedural Content Generation for Games](https://mlsanity.com/paper/2302.08479/) (2023)
- [From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priors](https://mlsanity.com/paper/2603.02792/) (2026)
- [PyTorch Hyperparameter Tuning - A Tutorial for spotPython](https://mlsanity.com/paper/2305.11930/) (2023)
- [PATH: A Discrete-sequence Dataset for Evaluating Online Unsupervised Anomaly Detection Approaches for Multivariate Time Series](https://mlsanity.com/paper/2411.13951/) (2024)
- [LLaMEA-SAGE: Guiding Automated Algorithm Design with Structural Feedback from Explainable AI](https://mlsanity.com/paper/2601.21511/) (2026)
- [Assessing Reproducibility in Evolutionary Computation: A Case Study using Human- and LLM-based Assessment](https://mlsanity.com/paper/2602.07059/) (2026)
- [Surrogate Model Based Hyperparameter Tuning for Deep Learning with SPOT](https://mlsanity.com/paper/2105.14625/) (2021)
- [OPTION: OPTImization Algorithm Benchmarking ONtology](https://mlsanity.com/paper/2211.11332/) (2022)
- [On the Algebraic Properties of Flame Graphs](https://mlsanity.com/paper/2301.08941/) (2023)
- [Time to Stop and Think: What kind of research do we want to do?](https://mlsanity.com/paper/2402.08298/) (2024)
- [Clustering in Dynamic Environments: A Framework for Benchmark Dataset Generation With Heterogeneous Changes](https://mlsanity.com/paper/2402.15731/) (2024)
- [A Novel Ranking Scheme for the Performance Analysis of Stochastic Optimization Algorithms using the Principles of Severity](https://mlsanity.com/paper/2406.00154/) (2024)
- [Generalizability of experimental studies](https://mlsanity.com/paper/2406.17374/) (2024)
- [Evolving Hard Maximum Cut Instances for Quantum Approximate Optimization Algorithms](https://mlsanity.com/paper/2502.12012/) (2025)

## Abstract (arXiv)

This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility. The final goal is to provide well-accepted guidelines (rules) that might be useful for authors and reviewers. As benchmarking in optimization is an active and evolving field of research this manuscript is meant to co-evolve over time by means of periodic updates.