# Benchmarking that Matters: Rethinking Benchmarking for Practical Impact

arXiv: https://arxiv.org/abs/2511.12264 · HTML: https://mlsanity.com/paper/2511.12264/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2025.

## Built on by (most cited first, 1 of 1)

- [On the Influence of the Feature Computation Budget on Per-Instance Algorithm Selection for Black-Box Optimization](https://mlsanity.com/paper/2605.04954/) (2026)

## Abstract (arXiv)

Benchmarking has driven scientific progress in Evolutionary Computation, yet current practices fall short of real-world needs. Widely used synthetic suites such as BBOB and CEC isolate algorithmic phenomena but poorly reflect the structure, constraints, and information limitations of continuous and mixed-integer optimization problems in practice. This disconnect leads to the misuse of benchmarking suites for competitions, automated algorithm selection, and industrial decision-making, despite these suites being designed for different purposes. We identify key gaps in current benchmarking practices and tooling, including limited availability of real-world-inspired problems, missing high-level features, and challenges in multi-objective and noisy settings. We propose a vision centered on curated real-world-inspired benchmarks, practitioner-accessible feature spaces and community-maintained performance databases. Real progress requires coordinated effort: A living benchmarking ecosystem that evolves with real-world insights and supports both scientific understanding and industrial use.