# Pitfalls and Best Practices in Algorithm Configuration

arXiv: https://arxiv.org/abs/1705.06058 · HTML: https://mlsanity.com/paper/1705.06058/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2017.

## Built on by (most cited first, 25 of 25)

- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [MLGym: A New Framework and Benchmark for Advancing AI Research Agents](https://mlsanity.com/paper/2502.14499/) (2025)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Hyperparameters in Reinforcement Learning and How To Tune Them](https://mlsanity.com/paper/2306.01324/) (2023)
- [A Survey of Methods for Automated Algorithm Configuration](https://mlsanity.com/paper/2202.01651/) (2022)
- [Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch](https://mlsanity.com/paper/2603.24647/) (2026)
- [Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization](https://mlsanity.com/paper/2405.15393/) (2024)
- [Taking the human out of decomposition-based optimization via artificial intelligence: Part II. Learning to initialize](https://mlsanity.com/paper/2310.07082/) (2023)
- [Combining Automated Optimisation of Hyperparameters and Reward Shape](https://mlsanity.com/paper/2406.18293/) (2024)
- [A Tutorial on the Design, Experimentation and Application of Metaheuristic Algorithms to Real-World Optimization Problems](https://mlsanity.com/paper/2410.03205/) (2024)
- [Accelerating process control and optimization via machine learning: A review](https://mlsanity.com/paper/2412.18529/) (2024)
- [Overtuning in Hyperparameter Optimization](https://mlsanity.com/paper/2506.19540/) (2025)
- [Assessing Ranking and Effectiveness of Evolutionary Algorithm Hyperparameters Using Global Sensitivity Analysis Methodologies](https://mlsanity.com/paper/2207.04820/) (2022)
- [carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks](https://mlsanity.com/paper/2506.06143/) (2025)
- [A learning-based mathematical programming formulation for the automatic configuration of optimization solvers](https://mlsanity.com/paper/2401.04237/) (2024)
- [The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning](https://mlsanity.com/paper/2407.18840/) (2024)
- [Performance Variation in Deep Reinforcement Learning](https://mlsanity.com/paper/2606.06746/) (2026)
- [An Automated Machine Learning (AutoML) Method for Driving Distraction Detection Based on Lane-Keeping Performance](https://mlsanity.com/paper/2103.08311/) (2021)
- [On the Configuration of More and Less Expressive Logic Programs](https://mlsanity.com/paper/2203.01024/) (2022)
- [Learning to Configure Mathematical Programming Solvers by Mathematical Programming](https://mlsanity.com/paper/2401.05041/) (2024)
- [Training neural networks faster with minimal tuning using pre-computed lists of hyperparameters for NAdamW](https://mlsanity.com/paper/2503.03986/) (2025)
- [(Machine) Learning to Improve the Empirical Performance of Discrete Algorithms](https://mlsanity.com/paper/2109.14271/) (2021)
- [Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Network](https://mlsanity.com/paper/2512.16491/) (2025)
- [Have I Solved This Before? Retrieving Similar Segmentation Problems for Evolutionary Learning](https://mlsanity.com/paper/2606.08155/) (2026)

## Abstract (arXiv)

Good parameter settings are crucial to achieve high performance in many areas of artificial intelligence (AI), such as propositional satisfiability solving, AI planning, scheduling, and machine learning (in particular deep learning). Automated algorithm configuration methods have recently received much attention in the AI community since they replace tedious, irreproducible and error-prone manual parameter tuning and can lead to new state-of-the-art performance. However, practical applications of algorithm configuration are prone to several (often subtle) pitfalls in the experimental design that can render the procedure ineffective. We identify several common issues and propose best practices for avoiding them. As one possibility for automatically handling as many of these as possible, we also propose a tool called GenericWrapper4AC.