mlsanityData to 7 Oct 2026

Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization

arXiv 2505.21321 ↗First posted 27 May 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We present Bencher, a modular benchmarking framework for black-box optimization that fundamentally decouples benchmark execution from optimization logic. Unlike prior suites that focus on combining many benchmarks in a single project, Bencher introduces a clean abstraction boundary: each benchmark is isolated in its own virtual Python environment and accessed via a unified, version-agnostic remote procedure call (RPC) interface. This design eliminates dependency conflicts and simplifies the integration of diverse, real-world benchmarks, which often have complex and conflicting software requirements. Bencher can be deployed locally or remotely via Docker or on high-performance computing (HPC) clusters via Singularity, providing a containerized, reproducible runtime for any benchmark. Its lightweight client requires minimal setup and supports drop-in evaluation of 80 benchmarks across continuous, categorical, and binary domains.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

24 of 33 references.

References · 24

  1. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
  2. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 146
  3. 2021High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspacescited by 139
  4. 2017Batched Large-scale Bayesian Optimization in High-dimensional Spacescited by 116
  5. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  6. 2020Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Searchcited by 87
  7. 2024Vanilla Bayesian Optimization Performs Great in High Dimensionscited by 87
  8. 2020Scalable Constrained Bayesian Optimizationcited by 84
  9. 2019Combinatorial Bayesian Optimization using the Graph Cartesian Productcited by 78
  10. 2017Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Spacecited by 76
  11. 2023Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspacescited by 50
  12. 2021LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lassocited by 41
12 more
  1. 2024Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimizationcited by 33
  2. 2022Local Bayesian optimization via maximizing probability of descentcited by 29
  3. 2018IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristicscited by 28
  4. 2023Bounce: Reliable High-Dimensional Bayesian Optimization for Combinatorial and Mixed Spacescited by 27
  5. 2021IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristicscited by 24
  6. 2023Bayesian Optimization over High-Dimensional Combinatorial Spaces via Dictionary-based Embeddingscited by 24
  7. 2025Understanding High-Dimensional Bayesian Optimizationcited by 19
  8. 2021Computationally Efficient High-Dimensional Bayesian Optimization via Variable Selectioncited by 12
  9. 2024Minimizing UCB: a Better Local Search Strategy in Local Bayesian Optimizationcited by 9
  10. 2020Benchmarking for Metaheuristic Black-Box Optimization: Perspectives and Open Challengescited by 4
  11. 2024CATBench: A Compiler Autotuning Benchmarking Suite for Black-box Optimizationcited by 2
  12. 2023High-Dimensional Bayesian Optimisation with Large-Scale Constraints -- An Application to Aeroelastic Tailoringcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priorscited by 2