# Simplifying Reward Design through Divide-and-Conquer

arXiv: https://arxiv.org/abs/1806.02501 · HTML: https://mlsanity.com/paper/1806.02501/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2018.

## Built on by (most cited first, 12 of 12)

- [Code as Reward: Empowering Reinforcement Learning with VLMs](https://mlsanity.com/paper/2402.04764/) (2024)
- [Direct Behavior Specification via Constrained Reinforcement Learning](https://mlsanity.com/paper/2112.12228/) (2021)
- [What are you optimizing for? Aligning Recommender Systems with Human Values](https://mlsanity.com/paper/2107.10939/) (2021)
- [Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learning](https://mlsanity.com/paper/2107.03961/) (2021)
- [Active Reward Learning from Online Preferences](https://mlsanity.com/paper/2302.13507/) (2023)
- [Policy Gradient Bayesian Robust Optimization for Imitation Learning](https://mlsanity.com/paper/2106.06499/) (2021)
- [Assisted Robust Reward Design](https://mlsanity.com/paper/2111.09884/) (2021)
- [LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learning](https://mlsanity.com/paper/2504.15472/) (2025)
- [Constrained Group Relative Policy Optimization](https://mlsanity.com/paper/2602.05863/) (2026)
- [Online Behavior Modification for Expressive User Control of RL-Trained Robots](https://mlsanity.com/paper/2408.16776/) (2024)
- [Effective Reward Specification in Deep Reinforcement Learning](https://mlsanity.com/paper/2412.07177/) (2024)
- [Supervised Reward Inference](https://mlsanity.com/paper/2502.18447/) (2025)

## Abstract (arXiv)

Designing a good reward function is essential to robot planning and reinforcement learning, but it can also be challenging and frustrating. The reward needs to work across multiple different environments, and that often requires many iterations of tuning. We introduce a novel divide-and-conquer approach that enables the designer to specify a reward separately for each environment. By treating these separate reward functions as observations about the underlying true reward, we derive an approach to infer a common reward across all environments. We conduct user studies in an abstract grid world domain and in a motion planning domain for a 7-DOF manipulator that measure user effort and solution quality. We show that our method is faster, easier to use, and produces a higher quality solution than the typical method of designing a reward jointly across all environments. We additionally conduct a series of experiments that measure the sensitivity of these results to different properties of the reward design task, such as the number of environments, the number of feasible solutions per environment, and the fraction of the total features that vary within each environment. We find that independent reward design outperforms the standard, joint, reward design process but works best when the design problem can be divided into simpler subproblems.