mlsanityData to 14 Oct 2026

Benchmarking in Optimization: Best Practice and Open Issues

arXiv 2007.03488 ↗First posted 7 Jul 2020cs.NE cs.PF math.OCcited by 38

Not on the map. Why?

Abstract (arXiv)

This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility. The final goal is to provide well-accepted guidelines (rules) that might be useful for authors and reviewers. As benchmarking in optimization is an active and evolving field of research this manuscript is meant to co-evolve over time by means of periodic updates.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 5; 2023: 5; 2024: 9; 2025: 8; 2026: 7.

38 papers.

Sort:

By area:

  1. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  2. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 112
  3. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
  4. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 41
  5. 2022Benchopt: Reproducible, efficient and collaborative optimization benchmarkscited by 27
  6. 2021Reproducibility in Evolutionary Computationcited by 13
  7. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 11
  8. 2024Explainable Benchmarking for Iterative Optimization Heuristicscited by 10
  9. 2025BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristicscited by 9
  10. 2022A Survey on Learnable Evolutionary Algorithms for Scalable Multiobjective Optimizationcited by 7
  11. 2025Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systemscited by 6
  12. 2022The Importance of Landscape Features for Performance Prediction of Modular CMA-ES Variantscited by 5
26 more
  1. 2023Using Automated Algorithm Configuration for Parameter Controlcited by 5
  2. 2022A Framework for Generating Informative Benchmark Instancescited by 4
  3. 2023Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gymcited by 4
  4. 2021Towards Standardising Reinforcement Learning Approaches for Production Scheduling Problemscited by 2
  5. 2023Tools for Landscape Analysis of Optimisation Problems in Procedural Content Generation for Gamescited by 2
  6. 2026From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priorscited by 2
  7. 2023PyTorch Hyperparameter Tuning - A Tutorial for spotPythoncited by 1
  8. 2024PATH: A Discrete-sequence Dataset for Evaluating Online Unsupervised Anomaly Detection Approaches for Multivariate Time Seriescited by 1
  9. 2026LLaMEA-SAGE: Guiding Automated Algorithm Design with Structural Feedback from Explainable AIcited by 1
  10. 2026Assessing Reproducibility in Evolutionary Computation: A Case Study using Human- and LLM-based Assessmentcited by 1
  11. 2021Surrogate Model Based Hyperparameter Tuning for Deep Learning with SPOTcited by 0
  12. 2022OPTION: OPTImization Algorithm Benchmarking ONtologycited by 0
  13. 2023On the Algebraic Properties of Flame Graphscited by 0
  14. 2024Time to Stop and Think: What kind of research do we want to do?cited by 0
  15. 2024Clustering in Dynamic Environments: A Framework for Benchmark Dataset Generation With Heterogeneous Changescited by 0
  16. 2024A Novel Ranking Scheme for the Performance Analysis of Stochastic Optimization Algorithms using the Principles of Severitycited by 0
  17. 2024Generalizability of experimental studiescited by 0
  18. 2025Evolving Hard Maximum Cut Instances for Quantum Approximate Optimization Algorithmscited by 0
  19. 2025Optimization Problem Solving Can Transition to Evolutionary Agentic Workflowscited by 0
  20. 2025Benchmarking MOEAs for solving continuous multi-objective RL problemscited by 0
  21. 2025From Performance to Understanding: A Vision for Explainable Automated Algorithm Designcited by 0
  22. 2025Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Networkcited by 0
  23. 2026Quantifying the Impact of Modules and Their Interactions in the PSO-X Frameworkcited by 0
  24. 2026Learning to Assess the Reliability of Number-of-Runs Estimation in Stochastic Optimizationcited by 0
  25. 2026From Hand-Crafted to LLM-Based Variation Operators in Metaheuristics: A Tutorialcited by 0
  26. 2026What a Reporting Convention Hides: A Matched-Budget Audit of Quantum Natural Gradient with an Exactly Computed Metriccited by 0