mlsanityData to 7 Oct 2026

COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting

arXiv 1603.08785 ↗First posted 29 Mar 2016cs.AI cs.MS cs.NAcited by 135

Not on the map. Why?

Abstract (arXiv)

We introduce COCO, an open source platform for Comparing Continuous Optimizers in a black-box setting. COCO aims at automatizing the tedious and repetitive task of benchmarking numerical optimization algorithms to the greatest possible extent. The platform and the underlying methodology allow to benchmark in the same framework deterministic and stochastic solvers for both single and multiobjective optimization. We present the rationales behind the (decade-long) development of the platform as a general proposition for guidelines towards better benchmarking. We detail underlying fundamental concepts of COCO such as the definition of a problem as a function instance, the underlying idea of instances, the use of target values, and runtime defined by the number of function calls as the central performance measure. Finally, we give a quick overview of the basic code structure and the currently available test suites.

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 24; 2023: 21; 2024: 19; 2025: 27; 2026: 28.

135 papers.

Sort:

By area:

  1. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 146
  2. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  3. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  4. 2021Differentiable Quality Diversitycited by 71
  5. 2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
  6. 2021YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimizationcited by 61
  7. 2022AMLB: an AutoML Benchmarkcited by 55
  8. 2021LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lassocited by 41
  9. 2023MetaBox: A Benchmark Platform for Meta-Black-Box Optimization with Reinforcement Learningcited by 35
  10. 2021TREGO: a Trust-Region Framework for Efficient Global Optimizationcited by 28
  11. 2021DACBench: A Benchmark Library for Dynamic Algorithm Configurationcited by 28
  12. 2022Benchopt: Reproducible, efficient and collaborative optimization benchmarkscited by 27
38 more
  1. 2022Covariance Matrix Adaptation MAP-Annealingcited by 24
  2. 2024Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimizationcited by 23
  3. 2023Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOBcited by 18
  4. 2023Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimizationcited by 15
  5. 2021Towards Explainable Exploratory Landscape Analysis: Extreme Feature Selection for Classifying BBOB Functionscited by 14
  6. 2022Accelerated Quality-Diversity through Massive Parallelismcited by 14
  7. 2025OptunaHub: A Platform for Black-Box Optimizationcited by 14
  8. 2021Meta Learning Black-Box Population-Based Optimizerscited by 13
  9. 2024Pretrained Optimization Model for Zero-Shot Black Box Optimizationcited by 13
  10. 2021Revisiting Bayesian Optimization in the light of the COCO benchmarkcited by 10
  11. 2024Explainable Benchmarking for Iterative Optimization Heuristicscited by 10
  12. 2024Neural Exploratory Landscape Analysis for Meta-Black-Box-Optimizationcited by 10
  13. 2025LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithmscited by 10
  14. 2023To Switch or not to Switch: Predicting the Benefit of Switching between Algorithms based on Trajectory Featurescited by 9
  15. 2025BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristicscited by 9
  16. 2025MetaBox-v2: A Unified Benchmark Platform for Meta-Black-Box Optimizationcited by 9
  17. 2025A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solvingcited by 9
  18. 2022A Collection of Deep Learning-based Feature-Free Approaches for Characterizing Single-Objective Continuous Fitness Landscapescited by 8
  19. 2023DynamoRep: Trajectory-Based Population Dynamics for Classification of Black-box Optimization Problemscited by 8
  20. 2024On the Utility of Probing Trajectories for Algorithm-Selectioncited by 8
  21. 2024Partial Rankings of Optimizerscited by 8
  22. 2022High Dimensional Bayesian Optimization with Kernel Principal Component Analysiscited by 7
  23. 2023MA-BBOB: Many-Affine Combinations of BBOB Functions for Evaluating AutoML Approaches in Noiseless Numerical Black-Box Optimization Contextscited by 7
  24. 2022PI is back! Switching Acquisition Functions in Bayesian Optimizationcited by 6
  25. 2026Adaptive Acquisition Selection for Bayesian Optimization with Large Language Modelscited by 6
  26. 2021Learning How to Optimize Black-Box Functions With Extreme Limits on the Number of Function Evaluationscited by 5
  27. 2021Explainable Landscape-Aware Optimization Performance Predictioncited by 5
  28. 2022The Importance of Landscape Features for Performance Prediction of Modular CMA-ES Variantscited by 5
  29. 2022Pre-training helps Bayesian optimization toocited by 5
  30. 2023MONGOOSE: Path-wise Smooth Bayesian Optimisation via Meta-learningcited by 5
  31. 2023Using Automated Algorithm Configuration for Parameter Controlcited by 5
  32. 2023Self-Adjusting Weighted Expected Improvement for Bayesian Optimizationcited by 5
  33. 2024Instance Selection for Dynamic Algorithm Configuration with Reinforcement Learning: Improving Generalizationcited by 5
  34. 2024Understanding Impacts of Electromagnetic Signal Injection Attacks on Object Detectioncited by 5
  35. 2025DesignX: Human-Competitive Algorithm Designer for Black-Box Optimizationcited by 5
  36. 2023Deep-BIAS: Detecting Structural Bias using Explainable AIcited by 4
  37. 2023Assessing the Generalizability of a Performance Predictive Modelcited by 4
  38. 2023On the Hyperparameter Loss Landscapes of Machine Learning Models: An Exploratory Studycited by 4

50 most cited of 135 All 135 (JSON)