# Hybrid Reward Architecture for Reinforcement Learning

arXiv: https://arxiv.org/abs/1706.04208 · HTML: https://mlsanity.com/paper/1706.04208/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2017.

## Built on by (most cited first, 30 of 58)

- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexity](https://mlsanity.com/paper/2210.09579/) (2022)
- [Reinforcement Learning, Bit by Bit](https://mlsanity.com/paper/2103.04047/) (2021)
- [Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare](https://mlsanity.com/paper/2305.01738/) (2023)
- [How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition](https://mlsanity.com/paper/2207.07730/) (2022)
- [Solving Continuous Control via Q-learning](https://mlsanity.com/paper/2210.12566/) (2022)
- [Distributional Reinforcement Learning for Multi-Dimensional Reward Functions](https://mlsanity.com/paper/2110.13578/) (2021)
- [Reinforcement learning on graphs: A survey](https://mlsanity.com/paper/2204.06127/) (2022)
- [AoI-Aware Resource Allocation for Platoon-Based C-V2X Networks via Multi-Agent Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2105.04196/) (2021)
- [Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learning](https://mlsanity.com/paper/2107.03961/) (2021)
- [Continuous Control with Coarse-to-fine Reinforcement Learning](https://mlsanity.com/paper/2407.07787/) (2024)
- [Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes](https://mlsanity.com/paper/2312.11118/) (2023)
- [Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agents](https://mlsanity.com/paper/2210.11825/) (2022)
- [Residual Q-Learning: Offline and Online Policy Customization without Value](https://mlsanity.com/paper/2306.09526/) (2023)
- [Asynchronous Hybrid Reinforcement Learning for Latency and Reliability Optimization in the Metaverse over Wireless Communications](https://mlsanity.com/paper/2212.14749/) (2022)
- [Distributional Reinforcement Learning with Regularized Wasserstein Loss](https://mlsanity.com/paper/2202.00769/) (2022)
- [Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution](https://mlsanity.com/paper/2404.04253/) (2024)
- [Reinforcement Learning Your Way: Agent Characterization through Policy Regularization](https://mlsanity.com/paper/2201.10003/) (2022)
- [Value Function Decomposition for Iterative Design of Reinforcement Learning Agents](https://mlsanity.com/paper/2206.13901/) (2022)
- [Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning](https://mlsanity.com/paper/2310.02360/) (2023)
- [Orchestrated Value Mapping for Reinforcement Learning](https://mlsanity.com/paper/2203.07171/) (2022)
- [Adapting to Reward Progressivity via Spectral Reinforcement Learning](https://mlsanity.com/paper/2104.14138/) (2021)
- [Locality Matters: A Scalable Value Decomposition Approach for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.10632/) (2021)
- [Flexible Multiple-Objective Reinforcement Learning for Chip Placement](https://mlsanity.com/paper/2204.06407/) (2022)
- [Towards model-free RL algorithms that scale well with unstructured data](https://mlsanity.com/paper/2311.02215/) (2023)
- [Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulations](https://mlsanity.com/paper/2506.16016/) (2025)
- [A Genetic Algorithm with Tree-structured Mutation for Hyperparameter Optimisation of Graph Neural Networks](https://mlsanity.com/paper/2102.11995/) (2021)
- [Learning robust driving policies without online exploration](https://mlsanity.com/paper/2103.08070/) (2021)
- [Decomposing the Prediction Problem; Autonomous Navigation by neoRL Agents](https://mlsanity.com/paper/2106.15868/) (2021)
- [Interpretable and Effective Reinforcement Learning for Attacking against Graph-based Rumor Detection](https://mlsanity.com/paper/2201.05819/) (2022)

## Abstract (arXiv)

One of the main challenges in reinforcement learning (RL) is generalisation. In typical deep RL methods this is achieved by approximating the optimal value function with a low-dimensional representation using a deep network. While this approach works well in many domains, in domains where the optimal value function cannot easily be reduced to a low-dimensional representation, learning can be very slow and unstable. This paper contributes towards tackling such challenging domains, by proposing a new method, called Hybrid Reward Architecture (HRA). HRA takes as input a decomposed reward function and learns a separate value function for each component reward function. Because each component typically only depends on a subset of all features, the corresponding value function can be approximated more easily by a low-dimensional representation, enabling more effective learning. We demonstrate HRA on a toy-problem and the Atari game Ms. Pac-Man, where HRA achieves above-human performance.