# Evolving Rewards to Automate Reinforcement Learning

arXiv: https://arxiv.org/abs/1905.07628 · HTML: https://mlsanity.com/paper/1905.07628/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2019.

## Built on by (most cited first, 30 of 31)

- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Preference Transformer: Modeling Human Preferences using Transformers for RL](https://mlsanity.com/paper/2303.00957/) (2023)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft](https://mlsanity.com/paper/2312.09238/) (2023)
- [Evolving Reinforcement Learning Algorithms](https://mlsanity.com/paper/2101.03958/) (2021)
- [Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Robotics](https://mlsanity.com/paper/2309.06687/) (2023)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimization](https://mlsanity.com/paper/2207.13676/) (2022)
- [AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning](https://mlsanity.com/paper/2301.10339/) (2023)
- [TTOpt: A Maximum Volume Quantized Tensor Train-based Optimization and its Application to Reinforcement Learning](https://mlsanity.com/paper/2205.00293/) (2022)
- [Reinforcement Learning with Automated Auxiliary Loss Search](https://mlsanity.com/paper/2210.06041/) (2022)
- [Automated Reinforcement Learning: An Overview](https://mlsanity.com/paper/2201.05000/) (2022)
- [Learning Synthetic Environments and Reward Networks for Reinforcement Learning](https://mlsanity.com/paper/2202.02790/) (2022)
- [Learning Robotic Manipulation Skills Using an Adaptive Force-Impedance Action Space](https://mlsanity.com/paper/2110.09904/) (2021)
- [Combining Automated Optimisation of Hyperparameters and Reward Shape](https://mlsanity.com/paper/2406.18293/) (2024)
- [RoboMoRe: LLM-based Robot Co-design via Joint Optimization of Morphology and Reward](https://mlsanity.com/paper/2506.00276/) (2025)
- [Deep Reinforcement Learning from Hierarchical Preference Design](https://mlsanity.com/paper/2309.02632/) (2023)
- [Discovering Minimal Reinforcement Learning Environments](https://mlsanity.com/paper/2406.12589/) (2024)
- [Impedance Primitive-augmented Hierarchical Reinforcement Learning for Sequential Tasks](https://mlsanity.com/paper/2508.19607/) (2025)
- [RDA: Reward Design Agent for Reinforcement Learning](https://mlsanity.com/paper/2606.01672/) (2026)
- [Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research](https://mlsanity.com/paper/2107.03015/) (2021)
- [Evolving Pareto-Optimal Actor-Critic Algorithms for Generalizability and Stability](https://mlsanity.com/paper/2204.04292/) (2022)
- [Continuously evolving rewards in an open-ended environment](https://mlsanity.com/paper/2405.01261/) (2024)
- [EvoRL: A GPU-accelerated Framework for Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2501.15129/) (2025)
- [SATA: Safe and Adaptive Torque-Based Locomotion Policies Inspired by Animal Learning](https://mlsanity.com/paper/2502.12674/) (2025)
- [An Automated Reinforcement Learning Reward Design Framework with Large Language Model for Cooperative Platoon Coordination](https://mlsanity.com/paper/2504.19480/) (2025)
- [A model-based framework for learning transparent swarm behaviors](https://mlsanity.com/paper/2103.05343/) (2021)
- [Robot Navigation Anticipative Strategies in Deep Reinforcement Motion Planning](https://mlsanity.com/paper/2210.08280/) (2022)
- [mathrmE2CFD: Towards Effective and Efficient Cost Function Design for Safe Reinforcement Learning via Large Language Model](https://mlsanity.com/paper/2407.05580/) (2024)
- [When to Localize? A Risk-Constrained Reinforcement Learning Approach](https://mlsanity.com/paper/2411.02788/) (2024)

## Abstract (arXiv)

Many continuous control tasks have easily formulated objectives, yet using them directly as a reward in reinforcement learning (RL) leads to suboptimal policies. Therefore, many classical control tasks guide RL training using complex rewards, which require tedious hand-tuning. We automate the reward search with AutoRL, an evolutionary layer over standard RL that treats reward tuning as hyperparameter optimization and trains a population of RL agents to find a reward that maximizes the task objective. AutoRL, evaluated on four Mujoco continuous control tasks over two RL algorithms, shows improvements over baselines, with the the biggest uplift for more complex tasks. The video can be found at: urlhttps://youtu.be/svdaOFfQyC8.