# On the Importance of Reward Design in Reinforcement Learning-based Dynamic Algorithm Configuration: A Case Study on OneMax with (1+(λ,λ))-GA

arXiv: https://arxiv.org/abs/2502.20265 · HTML: https://mlsanity.com/paper/2502.20265/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 26 of its 62 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [ParamILS: An Automatic Algorithm Configuration Framework](https://mlsanity.com/paper/1401.3492/) (2014)
- [Learning values across many orders of magnitude](https://mlsanity.com/paper/1602.07714/) (2016)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Automated Dynamic Algorithm Configuration](https://mlsanity.com/paper/2205.13881/) (2022)
- [Deep Reinforcement Learning Based Parameter Control in Differential Evolution](https://mlsanity.com/paper/1905.08006/) (2019)
- [A Survey of Methods for Automated Algorithm Configuration](https://mlsanity.com/paper/2202.01651/) (2022)
- [DACBench: A Benchmark Library for Dynamic Algorithm Configuration](https://mlsanity.com/paper/2105.08541/) (2021)
- [DORA The Explorer: Directed Outreaching Reinforcement Action-Selection](https://mlsanity.com/paper/1804.04012/) (2018)
- [Potential-Based Reward Shaping For Intrinsic Motivation](https://mlsanity.com/paper/2402.07411/) (2024)
- [Online Selection of CMA-ES Variants](https://mlsanity.com/paper/1904.07801/) (2019)
- [Theory-inspired Parameter Control Benchmarks for Dynamic Algorithm Configuration](https://mlsanity.com/paper/2202.03259/) (2022)
- [Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks](https://mlsanity.com/paper/2310.17805/) (2023)
- [Optimal Parameter Choices Through Self-Adjustment: Applying the 1/5-th Rule in Discrete Settings](https://mlsanity.com/paper/1504.03212/) (2015)
- [Using Automated Algorithm Configuration for Parameter Control](https://mlsanity.com/paper/2302.12334/) (2023)
- [Instance Selection for Dynamic Algorithm Configuration with Reinforcement Learning: Improving Generalization](https://mlsanity.com/paper/2407.13513/) (2024)
- [CANDID DAC: Leveraging Coupled Action Dimensions with Importance Differences in DAC](https://mlsanity.com/paper/2407.05789/) (2024)

## Built on by (most cited first, 5 of 5)

- [Multi-parameter Control for the (1+(λ,λ))-GA on OneMax via Deep Reinforcement Learning](https://mlsanity.com/paper/2505.12982/) (2025)
- [Deep Reinforcement Learning for Dynamic Algorithm Configuration: A Case Study on Optimizing OneMax with the (1+(λ,λ))-GA](https://mlsanity.com/paper/2512.03805/) (2025)
- [Reward Engineering for Spatial Epidemic Simulations: A Reinforcement Learning Platform for Individual Behavioral Learning](https://mlsanity.com/paper/2511.18000/) (2025)
- [READY: Reward Discovery for Meta-Black-Box Optimization](https://mlsanity.com/paper/2601.21847/) (2026)
- [Discovering Interpretable Multi-Parameter Control Policies for Evolutionary Algorithms Using Deep Reinforcement Learning](https://mlsanity.com/paper/2606.10129/) (2026)

## Abstract (arXiv)

Dynamic Algorithm Configuration (DAC) has garnered significant attention in recent years, particularly in the prevalence of machine learning and deep learning algorithms. Numerous studies have leveraged the robustness of decision-making in Reinforcement Learning (RL) to address the optimization challenges associated with algorithm configuration. However, making an RL agent work properly is a non-trivial task, especially in reward design, which necessitates a substantial amount of handcrafted knowledge based on domain expertise. In this work, we study the importance of reward design in the context of DAC via a case study on controlling the population size of the (1+(λ,λ))-GA optimizing OneMax. We observed that a poorly designed reward can hinder the RL agent's ability to learn an optimal policy because of a lack of exploration, leading to both scalability and learning divergence issues. To address those challenges, we propose the application of a reward shaping mechanism to facilitate enhanced exploration of the environment by the RL agent. Our work not only demonstrates the ability of RL in dynamically configuring the (1+(λ,λ))-GA, but also confirms the advantages of reward shaping in the scalability of RL agents across various sizes of OneMax problems.