# Reproducibility in Evolutionary Computation

arXiv: https://arxiv.org/abs/2102.03380 · HTML: https://mlsanity.com/paper/2102.03380/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 77 references that are in the map.
- First posted on arXiv 5 Feb 2021.

## Builds on (most cited first, 5 of 5)

- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [On the State of the Art of Evaluation in Neural Language Models](https://mlsanity.com/paper/1707.05589/) (2017)
- [Benchmarking in Optimization: Best Practice and Open Issues](https://mlsanity.com/paper/2007.03488/) (2020)
- [The Configurable SAT Solver Challenge (CSSC)](https://mlsanity.com/paper/1505.01221/) (2015)
- [Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances](https://mlsanity.com/paper/1908.01720/) (2019)

## Built on by (most cited first, 13 of 13)

- [BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristics](https://mlsanity.com/paper/2504.20183/) (2025)
- [Metaheuristics and Large Language Models Join Forces: Toward an Integrated Optimization Approach](https://mlsanity.com/paper/2405.18272/) (2024)
- [Using Automated Algorithm Configuration for Parameter Control](https://mlsanity.com/paper/2302.12334/) (2023)
- [Using Knowledge Graphs for Performance Prediction of Modular Optimization Algorithms](https://mlsanity.com/paper/2301.09876/) (2023)
- [Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym](https://mlsanity.com/paper/2310.00077/) (2023)
- [IterMask3D: Unsupervised Anomaly Detection and Segmentation with Test-Time Iterative Mask Refinement in 3D Brain MR](https://mlsanity.com/paper/2504.04911/) (2025)
- [The importance of being constrained: dealing with infeasible solutions in Differential Evolution and beyond](https://mlsanity.com/paper/2203.03512/) (2022)
- [Assessing Reproducibility in Evolutionary Computation: A Case Study using Human- and LLM-based Assessment](https://mlsanity.com/paper/2602.07059/) (2026)
- [Reliable validation of Reinforcement Learning Benchmarks](https://mlsanity.com/paper/2203.01075/) (2022)
- [OPTION: OPTImization Algorithm Benchmarking ONtology](https://mlsanity.com/paper/2211.11332/) (2022)
- [Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Network](https://mlsanity.com/paper/2512.16491/) (2025)
- [Metric Aggregation Divergence: A Hidden Validity Threat in Agent-Based Policy Optimization and a Contractual Remedy](https://mlsanity.com/paper/2606.29038/) (2026)
- [From Hand-Crafted to LLM-Based Variation Operators in Metaheuristics: A Tutorial](https://mlsanity.com/paper/2609.31649/) (2026)

## Abstract (arXiv)

Experimental studies are prevalent in Evolutionary Computation (EC), and concerns about the reproducibility and replicability of such studies have increased in recent times, reflecting similar concerns in other scientific fields. In this article, we discuss, within the context of EC, the different types of reproducibility and suggest a classification that refines the badge system of the Association of Computing Machinery (ACM) adopted by ACM Transactions on Evolutionary Learning and Optimization (https://dlnext.acm.org/journal/telo). We identify cultural and technical obstacles to reproducibility in the EC field. Finally, we provide guidelines and suggest tools that may help to overcome some of these reproducibility obstacles.